{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# H2O Uplift Distributed Random Forest \n",
    "\n",
    "### Author: Veronika Maurerova veronika.maurerova@h2o.ai\n",
    "\n",
    "## Modeling Uplift \n",
    "\n",
    "Distributed Uplift Random Forest (Uplift DRF) is a classification tool for modeling uplift - the incremental impact of a treatment. This tool is very useful for example in marketing or in medicine. This machine learning approach is inspired by the A/B testing method. \n",
    "\n",
    "To model uplift, the analyst needs to collect data specifically - before the experiment, the objects are divided usually into two groups: \n",
    "\n",
    "- **treatment group**: receive some kind of treatment (for example customer get some type of discount) \n",
    "- **control group**: is separated from the treatment (customers in this group get no discount). \n",
    "\n",
    "Then the data are prepared and an analyst can gather information about the response - for example, whether customers bought a product, patients recovered from the disease, or similar. \n",
    "\n",
    "## Uplift approaches \n",
    "\n",
    "There are several approaches to model uplift: \n",
    "\n",
    "- Meta-learner algorithms\n",
    "- Instrumental variables algorithms\n",
    "- Neural-networks-based algorithms\n",
    "- Tree-based algorithms \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tree Based Uplift Algorithm\n",
    "\n",
    "Tree-based algorithm means in every tree, it takes information about treatment/control group assignment and information about response directly into a decision about splitting a node. The uplift score is the criterion to make a decision similar to the Gini coefficient in the standard decision tree. \n",
    "\n",
    "**Uplift metric to decide best split**\n",
    "\n",
    "The goal is to maximize the differences between the class distributions in the treatment and control sets, so the splitting criteria are based on distribution divergences. The distribution divergence is calculated based on the ``uplift_metric`` parameter. In H2O-3, three ``uplift_metric`` types are supported:\n",
    "\n",
    "- **Kullback-Leibler divergence** (``uplift_metric=\"KL\"``) - uses logarithms to calculate divergence, asymmetric, widely used, tends to infinity values (if treatment or control group distributions contain zero values). \n",
    "\n",
    "$ KL(P, Q) = \\sum_{i=0}^{N} p_i \\log{\\frac{p_i}{q_i}}$\n",
    "\n",
    "- **Squared Euclidean distance** (``uplift_metric=\"euclidean\"``) - symmetric and stable distribution, does not tend to infinity values. \n",
    "\n",
    "$ E(P, Q) = \\sum_{i=0}^{N} (p_i-q_i)^2$\n",
    "\n",
    "\n",
    "- **Chi-squared divergence** (``uplift_metric=\"chi_squared\"``) - Euclidean divergence normalized by control group distribution. Asymmetric and also tends to infinity values (if control group distribution contains zero values). \n",
    "\n",
    "$X^2(P, Q) = \\sum_{i=0}^{N} \\frac{(p_i-q_i)^2}{q_i}$\n",
    "\n",
    "where:\n",
    "\n",
    "- $P$ is treatment group distribution\n",
    "\n",
    "- $Q$ is control group distribution\n",
    "\n",
    "In a tree node the result value for a split is sum: $metric(P, Q) + metric(1-P, 1-Q)$. \n",
    "\n",
    "For the split gain value, the result within the node is normalized using a Gini coefficient (Eclidean or ChiSquared) or entropy (KL) for each distribution before and after the split.\n",
    "\n",
    "\n",
    "**Uplift score in each leaf is calculated as:**\n",
    "\n",
    "- $TP = (TY1 + 1) / (T + 2)$\n",
    "- $CP = (CY1 + 1) / (C + 2)$\n",
    "- $uplift\\_score = TP - CP $\n",
    "\n",
    "where:\n",
    "- $T$ how many observations in a leaf are from the treatment group (how many data rows in a leaf have ``treatment_column`` label == 1) \n",
    "- $C$ how many observations in a leaf are from the control group (how many data rows in the leaf have ``treatment_column`` label == 0)\n",
    "- $TY1$ how many observations in a leaf are from the treatment group and respond to the offer (how many data rows in the leaf have ``treatment_column`` label == 1 and ``response_column`` label == 1)\n",
    "- $CY1$ how many observations in a leaf are from the control group and respond to the offer (how many data rows in the leaf have ``treatment_column`` label == 0 and ``response_column`` label == 1)\n",
    "\n",
    "**Note**: A higher uplift score means more observations from treatment group respond to the offer than from control group. Which means offered treatment has positive effect. The uplift score can be negative, if more observations from control group respond to the offer without treatment.\n",
    "\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "![Difference between SDT and Uplift DT](https://blog.h2o.ai/wp-content/uploads/2022/01/tree.png)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## H2O Implementation (Major release 3.36)\n",
    "\n",
    "The H2O-3 implementation of Uplift DRF is based on DRF because the principle of training is similar to DRF. It is tree based uplift algorithm. Uplift DRF generates a forest of classification uplift trees, rather than a single classification tree. Each of these trees is a weak learner built on a subset of rows and columns. More trees will reduce the variance. Classification takes the average prediction over all of their trees to make a final prediction. \n",
    "\n",
    "Currently, in H2O-3 only binomial trees are supported, as well as the uplift curve metric and Area Under Uplift curve (AUUC) metric, normalized AUUC, and the Qini value. We are working on adding also regression trees and more metrics, for example, Qini coefficient, and more. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Start H2O-3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import h2o\n",
    "from h2o.estimators.uplift_random_forest import H2OUpliftRandomForestEstimator\n",
    "\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.style as style\n",
    "\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load data\n",
    "\n",
    "To demonstrate how Uplift DRF works, Criteo dataset is used. \n",
    "\n",
    "**Source:**\n",
    "\n",
    "Diemert Eustache, Betlei Artem} and Renaudin, Christophe and Massih-Reza, Amini, \"A Large Scale Benchmark for Uplift Modeling\", ACM, Proceedings of the AdKDD and TargetAd Workshop, KDD, London,United Kingdom, August, 20, 2018, https://ailab.criteo.com/criteo-uplift-prediction-dataset/.\n",
    "\n",
    "\n",
    "\n",
    "**Description:**\n",
    "\n",
    "- The dataset was created by The Criteo AI Lab\n",
    "- Consists of 13M rows, each one representing a user with 12 features, a treatment indicator and 2 binary labels (visits and conversions).\n",
    "- Positive labels mean the user visited/converted on the advertiser website during the test period (2 weeks).\n",
    "- The global treatment ratio is 84.6%.\n",
    "\n",
    "**Detailed description of the columns:**\n",
    "\n",
    "- **f0, f1, f2, f3, f4, f5, f6, f7, f8, f9, f10, f11**: feature values (dense, float)\n",
    "- **treatment**: treatment group (1 = treated, 0 = control)\n",
    "- **conversion**: whether a conversion occured for this user (binary, label)\n",
    "- **visit**: whether a visit occured for this user (binary, label)\n",
    "- **exposure**: treatment effect, whether the user has been effectively exposed (binary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>f0</th>\n",
       "      <th>f1</th>\n",
       "      <th>f2</th>\n",
       "      <th>f3</th>\n",
       "      <th>f4</th>\n",
       "      <th>f5</th>\n",
       "      <th>f6</th>\n",
       "      <th>f7</th>\n",
       "      <th>f8</th>\n",
       "      <th>f9</th>\n",
       "      <th>f10</th>\n",
       "      <th>f11</th>\n",
       "      <th>treatment</th>\n",
       "      <th>conversion</th>\n",
       "      <th>visit</th>\n",
       "      <th>exposure</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>12.616365</td>\n",
       "      <td>10.059654</td>\n",
       "      <td>8.976429</td>\n",
       "      <td>4.679882</td>\n",
       "      <td>10.280525</td>\n",
       "      <td>4.115453</td>\n",
       "      <td>0.294443</td>\n",
       "      <td>4.833815</td>\n",
       "      <td>3.955396</td>\n",
       "      <td>13.190056</td>\n",
       "      <td>5.300375</td>\n",
       "      <td>-0.168679</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>12.616365</td>\n",
       "      <td>10.059654</td>\n",
       "      <td>9.002689</td>\n",
       "      <td>4.679882</td>\n",
       "      <td>10.280525</td>\n",
       "      <td>4.115453</td>\n",
       "      <td>0.294443</td>\n",
       "      <td>4.833815</td>\n",
       "      <td>3.955396</td>\n",
       "      <td>13.190056</td>\n",
       "      <td>5.300375</td>\n",
       "      <td>-0.168679</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>12.616365</td>\n",
       "      <td>10.059654</td>\n",
       "      <td>8.964775</td>\n",
       "      <td>4.679882</td>\n",
       "      <td>10.280525</td>\n",
       "      <td>4.115453</td>\n",
       "      <td>0.294443</td>\n",
       "      <td>4.833815</td>\n",
       "      <td>3.955396</td>\n",
       "      <td>13.190056</td>\n",
       "      <td>5.300375</td>\n",
       "      <td>-0.168679</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>12.616365</td>\n",
       "      <td>10.059654</td>\n",
       "      <td>9.002801</td>\n",
       "      <td>4.679882</td>\n",
       "      <td>10.280525</td>\n",
       "      <td>4.115453</td>\n",
       "      <td>0.294443</td>\n",
       "      <td>4.833815</td>\n",
       "      <td>3.955396</td>\n",
       "      <td>13.190056</td>\n",
       "      <td>5.300375</td>\n",
       "      <td>-0.168679</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>12.616365</td>\n",
       "      <td>10.059654</td>\n",
       "      <td>9.037999</td>\n",
       "      <td>4.679882</td>\n",
       "      <td>10.280525</td>\n",
       "      <td>4.115453</td>\n",
       "      <td>0.294443</td>\n",
       "      <td>4.833815</td>\n",
       "      <td>3.955396</td>\n",
       "      <td>13.190056</td>\n",
       "      <td>5.300375</td>\n",
       "      <td>-0.168679</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          f0         f1        f2        f3         f4        f5        f6  \\\n",
       "0  12.616365  10.059654  8.976429  4.679882  10.280525  4.115453  0.294443   \n",
       "1  12.616365  10.059654  9.002689  4.679882  10.280525  4.115453  0.294443   \n",
       "2  12.616365  10.059654  8.964775  4.679882  10.280525  4.115453  0.294443   \n",
       "3  12.616365  10.059654  9.002801  4.679882  10.280525  4.115453  0.294443   \n",
       "4  12.616365  10.059654  9.037999  4.679882  10.280525  4.115453  0.294443   \n",
       "\n",
       "         f7        f8         f9       f10       f11  treatment  conversion  \\\n",
       "0  4.833815  3.955396  13.190056  5.300375 -0.168679          1           0   \n",
       "1  4.833815  3.955396  13.190056  5.300375 -0.168679          1           0   \n",
       "2  4.833815  3.955396  13.190056  5.300375 -0.168679          1           0   \n",
       "3  4.833815  3.955396  13.190056  5.300375 -0.168679          1           0   \n",
       "4  4.833815  3.955396  13.190056  5.300375 -0.168679          1           0   \n",
       "\n",
       "   visit  exposure  \n",
       "0      0         0  \n",
       "1      0         0  \n",
       "2      0         0  \n",
       "3      0         0  \n",
       "4      0         0  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "control_name = \"control\"\n",
    "treatment_column = \"treatment\"\n",
    "response_column = \"visit\"\n",
    "feature_cols = [\"f\"+str(x) for x in range(0,12)]\n",
    "\n",
    "df = pd.read_csv(\"../../bigdata/server/criteo-uplift-v2.1.csv\")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare data\n",
    "\n",
    "Inspiration from: https://www.kaggle.com/code/hughhuyton/criteo-uplift-modelling/notebook\n",
    "\n",
    "To modeling uplift the treatment and control group data have to have similar distribution. In real world usually the control group is smaller than the treatment group. It is also a case of Crieteo dataset and we have to rebalanced the data to have a similar size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of samples: 13979592\n",
      "The dataset is largely imbalanced: \n",
      "1    0.85\n",
      "0    0.15\n",
      "Name: treatment, dtype: float64\n",
      "Percentage of users that visit: 4.7%\n",
      "Percentage of users that convert: 0.29%\n",
      "Percentage of visitors that convert: 6.21%\n"
     ]
    }
   ],
   "source": [
    "print('Total number of samples: {}'.format(len(df)))\n",
    "print('The dataset is largely imbalanced: ')\n",
    "print(df['treatment'].value_counts(normalize = True))\n",
    "print('Percentage of users that visit: {}%'.format(100*round(df['visit'].mean(),4)))\n",
    "print('Percentage of users that convert: {}%'.format(100*round(df['conversion'].mean(),4)))\n",
    "print('Percentage of visitors that convert: {}%'.format(100*round(df[df[\"visit\"]==1][\"conversion\"].mean(),4)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Class 0: 2096937\n",
      "Class 1: 11882655\n",
      "Proportion: 6 : 1\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlsAAAF6CAYAAAAu3Z1/AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAco0lEQVR4nO3de5SlVX3m8e/DRQGFtiONiDK0YpRBxzhjuxRHBy8otpdg8IJBswYHV4+zRp14v8S1bPES0BEykWSAiDgqygAiBhXl4oB3kyZeErGJGBGEiI1Wc7ERkf7NH+9b4eRQ3fVWd++uOtXfz1pnnTp7v/t9f+cUq3h6v7t2paqQJElSGzvNdwGSJEmLmWFLkiSpIcOWJElSQ4YtSZKkhgxbkiRJDRm2JEmSGjJsSZpoSZYnqSS7zHct05K8NMlF2/B830/ylP7r1Uk+vg3P/bYkH9pW55N0T4YtaYFKctvIY2OS20dev3QbXucjSd69rc43x2sPCkpJHp7knCQ3Jbk5yfeSvC7Jztur1pFaPpLkN0lu7R//kORPkyyZPqaqzqyqZw4816yffVU9sqou28rSSfKUJD8dO/d7q+oVW3tuSZtm2JIWqKq67/QDuBZ43kjbmdPHLaQZnRaSHAh8C7gO+HdVtQR4EbAC2HOeynpfVe0JLANeDjwB+FqS+2zLiyz27620ozBsSRNmenYiyZuT/Aw4I8lOSd6S5EdJfpHk7CS/MzLmnCQ/62eFvpzkkX37KuClwJv6GbML+vZrkryxn0H6VZLTkzwgyYX9bM4lSZaOnP8JSb6eZH2S707f8ur7LkvyriRf68delGTvvvvL/fP6/vqHzPCW3wl8vapeV1X/DFBVV1XV0VW1fobP5+VJftBf65+S/NeRvr2TfLav85dJvpJkp77vzUmu78ddleTps30vqurXVfW3wO8D96cLXiQ5JslX+6+T5KQkP09yS5K/T/KoWT77Nyf5HvCrJLv0bYeNXHq3JP+3r/XvkvzeyHusJA8bef2RJO/ug+CFwH4jM6T7Zey2ZJLfT3fbcn3/vfu3I33XJHlD/9/FzX0Nu832OUk7OsOWNJn2BX4HOABYBbwaeD5wKLAfMAX8xcjxFwK/C+wD/B1wJkBVndZ//b5+xux5I2NeADwDeDjwvP4cb6ObzdkJeA1AkgcBnwPe3df0BuBTSZaNnOtouiCyD3Cv/hiA/9Q/36+//jdmeK+HAecO+1gA+DnwXGCv/ponJfkPfd/rgZ/27+EB/fupJI8AXgU8rp+xOhy4ZugFq+pW4GLgyTN0P5PufT4cWAK8GPjFLJ/9HwLPoftcfjvDOY8AzqH7vD8BnJ9k11lq/BWwErhhZIb0htFjkjwc+CTwx3Sf0eeBC5Lca+SwFwPPAh4CPBo4ZnPXlbTAw1aSD/f/GvyHAceelOQ7/eMfk6zfDiVK82Uj8I6quqOqbgdeCfxJVf20qu4AVgMvnL4NVVUfrqpbR/p+LyNrjDbhg1V1Y1VdD3wF+FZVfbuqfg18Gvj3/XEvAz5fVZ+vqo1VdTGwBnj2yLnOqKp/7Gs9G3jMHN7r/YF/HnpwVX2uqn5UncuBi7g7BN0JPBA4oKrurKqvVPcHYu8C7g0cnGTXqrqmqn40hxoBbqALP+PupLvdeRCQqvrB9AzdZvx5VV3Xf14zuaKqzq2qO4ETgd3obmVuraOAz1XVxf25/yewO/DEsdpuqKpfAhcwt++ltENa0GEL+Ajdv6BmVVWvrarHVNVjgA8C5zWsS5pv6/rQM+0A4NP9rZ/1wA/oAsQDkuyc5Pj+FuMt3D1jszebd+PI17fP8Pq+I9d+0fS1++s/iS7UTPvZyNcbRsYO8Yuxc21WkpVJvtnfJlxPF/qm3+v7gauBi/pbjG8BqKqr6WZzVgM/T3JWkv3mUCPAg4BfjjdW1ZeAk+lmGn+e5LQke81yruuG9lfVRrrZurnWO5P9gJ+Mnfs6uvc2bWu+l9IOaUGHrar6MmM/vJIcmOQLSa7o11scNMPQP6SbCpcWqxp7fR2wsqruN/LYrZ+VOpruttNhdLexlvdjsolzzdV1wMfGrn2fqjp+C97HTC6hu6U5qyT3Bj5FNyPzgKq6H92tsEB3u6+qXl9VD6VbZ/W66bVZVfWJqnoSXXgs4IQh1+yve1+6z/crM/VX1Z9X1WOBg+luJ75xumsTp5ztc9l/5No7AQ+mm1mDLgDtMXLsvnM47w1073/63Omvdf0s4yRtxoIOW5twGvDq/gfXG4C/HO1McgDdWoIvzUNt0nw5BXhP/98/SZYlOaLv2xO4g26GaA/gvWNjbwQeuhXX/jjwvCSH97Nou6VbxP/gAWPX0d0S3dz13wE8Mcn7k+wLkORhST6e5H5jx96L7nbgOuC3SVbSrZmiH/fcfmyAm+lm/zYmeUSSp/Vh7dd0M3cbZys+yb2TPBY4n26d3BkzHPO4JI/v11T9qj//9Lm39LN/bJIj+9vEf0z3/f1m3/cd4Oj+e/EsunV8024E7r+ZW8hnA89J8vS+3tf35/76FtQoqTdRYav/1+MTgXOSfAc4lXveXngJcG5V3bWdy5Pm0/8C/pru9titdP/jfXzf91G6W0PXA1dy9/+Up51Ot1ZpfZLz53rhqrqObubsbXQh5zq6mZtZf75U1QbgPXTbJqxPco91R/3aqUPoZuS+n+RmutmrNcCtY8feSrdw/2y68HM03ecy7XfpZspuA74B/GVV/T+6gHY8cBPdbbJ9gLdupvQ39Z/zL+g+3yuAJ/aL0MftBfxVX89P+jHv7/u29LP/DN36qingj4Aj+zVWAP+D7hca1tP9tuO/nLeq1tLN+v9Tf81/deuxqq6iW4P3QbrP4nl0W478Zg61SRqTbm3owpVkOfDZqnpUv87hqqra5PqNJN8G/ntV+S8xSZI07yZqZquqbgF+nORF8C/714zuL3MQsJTuX6ySJEnzbkGHrSSfpAtOj0i3ieOxdNPixyb5LvB9utsX014CnFULfbpOkiTtMBb8bURJkqRJtqBntiRJkiadYUuSJKmhBfsX5ffee+9avnz5fJchSZI0qyuuuOKmqlo2U9+CDVvLly9nzZo1812GJEnSrJL8ZFN93kaUJElqyLAlSZLUkGFLkiSpIcOWJElSQ4YtSZKkhgaFrSQPS3Jqku8luSvJZQPGPC7JGUmuTrIhyVVJ3pFkt62uWpIkaUIM3frhkcCzgW8Cuw4ccxRwIHAC8EPg0cC7+ucXzK1MSZKkyTQ0bF1QVZ8BSHIusPeAMcdX1U0jry9L8mvg1CQHVNUm96OQJElaLAbdRqyqjXM98VjQmvbt/nm/uZ5PkiRpEm3vBfKHABuBH23n60qSJM2L7Ra2kuwLvB34WFX9fBPHrEqyJsmadevWba/SJEmSmtkuYSvJvYCzgduA127quKo6rapWVNWKZctm/FuOkiRJE6X5H6JOEuCjdL/R+B+raqr1NSVJkhaK5mEL+DPgCOAZVbV2O1xvUXvIWz433yVoQvz4+OfMdwmSJBqHrSRvBV4FvLiqvtryWpIkSQvRoLCVZA+6TU0BHgTsleSF/evPV9WGJFcDl1fVsf2Yo4H3Ah8Brk/yhJFT/qiqXAEvSZIWvaEzW/sA54y1Tb9+CHBNf66dR/qf2T8f0z9GvZwuhEmSJC1qg8JWVV0DZJZjlo+9PoZ7hixJkqQdyvbe1FSSJGmHYtiSJElqyLAlSZLUkGFLkiSpIcOWJElSQ4YtSZKkhgxbkiRJDRm2JEmSGjJsSZIkNWTYkiRJasiwJUmS1JBhS5IkqSHDliRJUkOGLUmSpIYMW5IkSQ0ZtiRJkhoybEmSJDVk2JIkSWrIsCVJktSQYUuSJKkhw5YkSVJDhi1JkqSGDFuSJEkNGbYkSZIaMmxJkiQ1ZNiSJElqyLAlSZLUkGFLkiSpIcOWJElSQ4YtSZKkhgxbkiRJDRm2JEmSGjJsSZIkNWTYkiRJasiwJUmS1JBhS5IkqSHDliRJUkOGLUmSpIYMW5IkSQ0NCltJHpbk1CTfS3JXkssGjluS5IwkU0luTnJmkvtvVcWSJEkTZJeBxz0SeDbwTWDXOZz/bODhwCuAjcAJwPnAk+dwDkmSpIk1NGxdUFWfAUhyLrD3bAOSHAI8Ezi0qr7ct10PfCvJYVV1yRbWLEmSNDEG3Uasqo1bcO6VwI3TQas/z98AP+77JEmSFr2WC+QPAtbO0P6Dvk+SJGnRaxm2lgLrZ2if6vvuIcmqJGuSrFm3bl3D0iRJkraPBbX1Q1WdVlUrqmrFsmXL5rscSZKkrdYybE0BS2ZoX9r3SZIkLXotw9ZaZl6btam1XJIkSYtOy7B1IbBvkidNNyRZATy075MkSVr0Bu2zlWQPuk1NAR4E7JXkhf3rz1fVhiRXA5dX1bEAVfWNJBcBH03yBu7e1PSr7rElSZJ2FEM3Nd0HOGesbfr1Q4Br+nPtPHbMUcBJwIfpZtE+C7xmSwqVJEmaRIPCVlVdA2SWY5bP0LYeeHn/kCRJ2uEsqK0fJEmSFhvDliRJUkOGLUmSpIYMW5IkSQ0ZtiRJkhoybEmSJDVk2JIkSWrIsCVJktSQYUuSJKkhw5YkSVJDhi1JkqSGDFuSJEkNGbYkSZIaMmxJkiQ1ZNiSJElqyLAlSZLUkGFLkiSpIcOWJElSQ4YtSZKkhgxbkiRJDRm2JEmSGjJsSZIkNWTYkiRJasiwJUmS1JBhS5IkqSHDliRJUkOGLUmSpIYMW5IkSQ0ZtiRJkhoybEmSJDVk2JIkSWrIsCVJktSQYUuSJKkhw5YkSVJDhi1JkqSGDFuSJEkNGbYkSZIaMmxJkiQ1ZNiSJElqaFDYSnJwkkuTbEhyQ5Ljkuw8YNyKJBcl+WX/uCTJ47e+bEmSpMkwa9hKshS4BCjgCOA44PXAO2cZt38/bhfgj/rHLsDFSQ7YurIlSZImwy4DjnklsDtwZFXdQheW9gJWJ3lf3zaT5wB7An9QVTcDJPk6cBPwbOB/b3X1kiRJC9yQ24grgS+Ohaqz6ALYoZsZtyvwW+BXI2239W2ZY52SJEkTaUjYOghYO9pQVdcCG/q+TflUf8wHkuyTZB/gJGAKOGfLypUkSZosQ8LWUmD9DO1Tfd+MquoG4KnAC4Ab+8eRwOFVtW7OlUqSJE2gZls/JHkg3QzWFXS3Ilf2X38uyb/ZxJhVSdYkWbNunXlMkiRNviFhawpYMkP70r5vU95It27rhVX1har6At0s113AG2YaUFWnVdWKqlqxbNmyAaVJkiQtbEPC1lrG1mb12zrswdharjEHAd+vqjunG6rqN8D3gQPnXqokSdLkGRK2LgQOT7LnSNtRwO3A5ZsZ9xPgUUnuNd2Q5N7Ao4Br5l6qJEnS5BkStk4B7gDOS3JYklXAauDE0e0gklyd5PSRcR8C9gM+neQ5SZ4LnA88EDhtG9UvSZK0oM0atqpqCng6sDNwAd3O8ScB7xg7dJf+mOlxVwDPotvY9GPAR+luPT6jqr67LYqXJEla6IbsIE9VXQk8bZZjls/Qdilw6RZVJkmStAg02/pBkiRJhi1JkqSmDFuSJEkNGbYkSZIaMmxJkiQ1ZNiSJElqyLAlSZLUkGFLkiSpIcOWJElSQ4YtSZKkhgxbkiRJDRm2JEmSGjJsSZIkNWTYkiRJasiwJUmS1JBhS5IkqSHDliRJUkOGLUmSpIYMW5IkSQ0ZtiRJkhoybEmSJDVk2JIkSWrIsCVJktSQYUuSJKkhw5YkSVJDhi1JkqSGDFuSJEkNGbYkSZIaMmxJkiQ1ZNiSJElqyLAlSZLUkGFLkiSpIcOWJElSQ4YtSZKkhgxbkiRJDRm2JEmSGjJsSZIkNWTYkiRJasiwJUmS1JBhS5IkqaFBYSvJwUkuTbIhyQ1Jjkuy88CxRyb52yS3J/lFki8kuc/WlS1JkjQZZg1bSZYClwAFHAEcB7weeOeAsa8APgFcCKwEXgH8ENhly0uWJEmaHENCzyuB3YEjq+oW4OIkewGrk7yvb7uHJHsDJwGvrqq/Gun69NYWLUmSNCmG3EZcCXxxLFSdRRfADt3MuBf3z/9nC2uTJEmaeEPC1kHA2tGGqroW2ND3bcrjgauAY5P8NMmdSb6V5IlbXK0kSdKEGRK2lgLrZ2if6vs2ZV/gEcDbgTcDzwN+BXwhyQNmGpBkVZI1SdasW7duQGmSJEkLW8utHwLcFzi2qs6sqi8AzwfuAl4104CqOq2qVlTVimXLljUsTZIkafsYEramgCUztC/t+zY3roDLphv6dV9XAAcPL1GSJGlyDQlbaxlbm5Vkf2APxtZyjfkB3exWxtoDbJxDjZIkSRNrSNi6EDg8yZ4jbUcBtwOXb2bcZ/vnp043JFkCPBb47hzrlCRJmkhDwtYpwB3AeUkOS7IKWA2cOLodRJKrk5w+/bqq1gCfAU5P8p+TPAf4a+BO4C+24XuQJElasGYNW1U1BTwd2Bm4gG7n+JOAd4wdukt/zKiXAecDJwLn0gWtp/XnlCRJWvQG/dmcqroSeNosxyyfoe024L/1D0mSpB1Oy60fJEmSdniGLUmSpIYMW5IkSQ0ZtiRJkhoybEmSJDVk2JIkSWrIsCVJktSQYUuSJKkhw5YkSVJDhi1JkqSGDFuSJEkNGbYkSZIaMmxJkiQ1ZNiSJElqyLAlSZLUkGFLkiSpIcOWJElSQ4YtSZKkhgxbkiRJDRm2JEmSGjJsSZIkNWTYkiRJasiwJUmS1JBhS5IkqSHDliRJUkOGLUmSpIYMW5IkSQ0ZtiRJkhoybEmSJDVk2JIkSWrIsCVJktSQYUuSJKkhw5YkSVJDhi1JkqSGDFuSJEkNGbYkSZIaMmxJkiQ1ZNiSJElqyLAlSZLU0KCwleTgJJcm2ZDkhiTHJdl56EWS7JRkTZJK8twtL1eSJGmy7DLbAUmWApcAVwJHAAcCH6ALam8feJ1XAA/ewholSZIm1pCZrVcCuwNHVtXFVXUK8E7gdUn2mm1wH9beA/zJVlUqSZI0gYaErZXAF6vqlpG2s+gC2KEDxr8L+Bpw6dzLkyRJmmxDwtZBwNrRhqq6FtjQ921SkkcD/wV4w5YWKEmSNMmGhK2lwPoZ2qf6vs35IHByVV09pJgkq/qF9GvWrVs3ZIgkSdKC1mzrhyQvAR4BvHvomKo6rapWVNWKZcuWtSpNkiRpuxkStqaAJTO0L+377iHJrsD7gROAnZLcD5heTH+fJHvOvVRJkqTJMyRsrWVsbVaS/YE9GFvLNeI+dFs9nEgXyKaA7/Z9ZwHf3pJiJUmSJs2s+2wBFwJvTLJnVd3atx0F3A5cvokxtwFPHWvbF/gk8DbgS1tQqyRJ0sQZErZOAV4DnJfkBOChwGrgxNHtIJJcDVxeVcdW1W+By0ZPkmR5/+XfV9W3tr50SZKkhW/WsFVVU0meDpwMXED3m4kn0QWu8XMN/hM+kiRJO4IhM1tU1ZXA02Y5Zvks/dcAGVqYJEnSYtBs6wdJkiQZtiRJkpoybEmSJDVk2JIkSWrIsCVJktSQYUuSJKkhw5YkSVJDhi1JkqSGDFuSJEkNGbYkSZIaMmxJkiQ1ZNiSJElqyLAlSZLUkGFLkiSpIcOWJElSQ4YtSZKkhgxbkiRJDRm2JEmSGjJsSZIkNWTYkiRJasiwJUmS1JBhS5IkqSHDliRJUkOGLUmSpIYMW5IkSQ0ZtiRJkhoybEmSJDVk2JIkSWrIsCVJktSQYUuSJKkhw5YkSVJDhi1JkqSGDFuSJEkNGbYkSZIaMmxJkiQ1ZNiSJElqyLAlSZLUkGFLkiSpIcOWJElSQ4PCVpKDk1yaZEOSG5Icl2TnWcY8LskZSa7ux12V5B1Jdts2pUuSJC18u8x2QJKlwCXAlcARwIHAB+iC2ts3M/So/tgTgB8Cjwbe1T+/YKuqliRJmhCzhi3glcDuwJFVdQtwcZK9gNVJ3te3zeT4qrpp5PVlSX4NnJrkgKr6ydaVLkmStPANCVsrgS+Ohaqz6GasDgUumGnQWNCa9u3+eT/AsCVJC8XqJfNdgSbF6pvnu4KJM2TN1kHA2tGGqroW2ND3zcUhwEbgR3McJ0mSNJGGhK2lwPoZ2qf6vkGS7Eu3xutjVfXzoeMkSZIm2XbZ+iHJvYCzgduA127muFVJ1iRZs27duu1RmiRJUlNDwtYUMNPN/KV932YlCfBR4JHAs6tqk2Oq6rSqWlFVK5YtWzagNEmSpIVtyAL5tYytzUqyP7AHY2u5NuHP6LaMeEZVDTlekiRp0Rgys3UhcHiSPUfajgJuBy7f3MAkbwVeBbysqr66xVVKkiRNqCFh6xTgDuC8JIclWQWsBk4c3Q6i3yn+9JHXRwPvpbuFeH2SJ4w8vEcoSZJ2CLPeRqyqqSRPB06m21NrPXASXeAaP9fon/B5Zv98TP8Y9XLgI3OsVZIkaeIMWbNFVV0JPG2WY5aPvT6Ge4YsSZKkHcp22fpBkiRpR2XYkiRJasiwJUmS1JBhS5IkqSHDliRJUkOGLUmSpIYMW5IkSQ0ZtiRJkhoybEmSJDVk2JIkSWrIsCVJktSQYUuSJKkhw5YkSVJDhi1JkqSGDFuSJEkNGbYkSZIaMmxJkiQ1ZNiSJElqyLAlSZLUkGFLkiSpIcOWJElSQ4YtSZKkhgxbkiRJDRm2JEmSGjJsSZIkNWTYkiRJasiwJUmS1JBhS5IkqSHDliRJUkOGLUmSpIYMW5IkSQ0ZtiRJkhoybEmSJDVk2JIkSWrIsCVJktSQYUuSJKkhw5YkSVJDhi1JkqSGDFuSJEkNDQpbSQ5OcmmSDUluSHJckp0HjFuS5IwkU0luTnJmkvtvfdmSJEmTYZfZDkiyFLgEuBI4AjgQ+ABdUHv7LMPPBh4OvALYCJwAnA88eYsrliRJmiCzhi3glcDuwJFVdQtwcZK9gNVJ3te33UOSQ4BnAodW1Zf7tuuBbyU5rKou2TZvQZIkaeEachtxJfDFsVB1Fl0AO3SWcTdOBy2Aqvob4Md9nyRJ0qI3JGwdBKwdbaiqa4ENfd/gcb0fzDJOkiRp0RgStpYC62don+r7tvU4SZKkRWPImq3tJskqYFX/8rYkV81nPZooewM3zXcRC0lOmO8KpEXBny3j3pn5rmChOmBTHUPC1hSwZIb2pX3f5sYtm8u4qjoNOG1ATdK/kmRNVa2Y7zokLS7+bNG2MOQ24lrG1lgl2R/Yg5nXZG1yXG9Ta7kkSZIWnSFh60Lg8CR7jrQdBdwOXD7LuH2TPGm6IckK4KF9nyRJ0qI3JGydAtwBnJfksH5d1WrgxNHtIJJcneT06ddV9Q3gIuCjSY5M8nzgTOCr7rGlBrz9LKkFf7Zoq6WqZj8oORg4GTiE7jcMPwSsrqq7Ro65Brisqo4ZabsfcBLwB3TB7rPAa6rKxYaSJGmHMChsSZIkacsM+kPU0kKU5GFJTk3yvSR3JblsvmuSNPmSHJzk0iQbktyQ5LgkO893XZpcC2qfLWmOHgk8G/gmsOs81yJpEUiyFLgEuBI4AjgQ+ADd5MTb57E0TTBvI2piJdmpqjb2X58L7F1VT5nfqiRNsiRvBd4EHDD9S2BJ3kT3i2H7jv2dYGkQbyNqYk0HLUnahlYCXxwLVWcBuwOHzk9JmnSGLUmS7naPjber6lpgAzNv1C3NyrAlSdLdltJtcTRuqu+T5sywJUmS1JBhS5Kku00BS2ZoX9r3SXNm2JIk6W5rGVublWR/YA/G1nJJQxm2JEm624XA4Un2HGk7CrgduHx+StKkc1NTTawke9BtagrwIGCvJC/sX3++qjbMT2WSJtgpwGuA85KcADyUbo+tE91jS1vKTU01sZIsB368ie6HVNU1268aSYtFkoOBk4FD6H4z8UPA6qq6az7r0uQybEmSJDXkmi1JkqSGDFuSJEkNGbYkSZIaMmxJkiQ1ZNiSJElqyLAlSZLUkGFLkiSpIcOWJElSQ4YtSZKkhv4/7K3m0HMYDecAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Print proportion of a binary column\n",
    "# https://www.kaggle.com/code/hughhuyton/criteo-uplift-modelling/notebook\n",
    "def print_proportion(df, column):\n",
    "    fig = plt.figure(figsize = (10,6))\n",
    "    target_count = df[column].value_counts()\n",
    "    print('Class 0:', target_count[0])\n",
    "    print('Class 1:', target_count[1])\n",
    "    print('Proportion:', int(round(target_count[1] / target_count[0])), ': 1')\n",
    "    target_count.plot(kind='bar', title='Treatment Class Distribution', color=['#2077B4', '#FF7F0E'], fontsize = 15)\n",
    "    plt.xticks(rotation=0) \n",
    "    \n",
    "print_proportion(df, treatment_column)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(11183673, 16)\n",
      "(2795919, 16)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "train_df, test_df  = train_test_split(df, test_size=0.2, random_state=42, stratify=df['treatment'])\n",
    "print(train_df.shape)\n",
    "print(test_df.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "del(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Class 0: 1677550\n",
      "Class 1: 9506123\n",
      "Proportion: 6 : 1\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAk0AAAF6CAYAAAAEbWzQAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAVsklEQVR4nO3debCld13n8c83C0uEBEgamCCk2QKCDo7EQhhFEIY9YKEshpkpUCfDVLkNW4CZGoOCsogU68RWIKVsArInUTbZBKzpsCmBMCzBkADphHQWEgKS7/zxPBfuXLpzf52+p+9J9+tV1dX3nuec5/nec1M37/49zz2nujsAAFyzgzZ7AACA6wLRBAAwQDQBAAwQTQAAA0QTAMAA0QQAMEA0AUuhqrZWVVfVIZs9y4qqelxVvXsD9/fZqrrP/PHJVfWaDdz3M6vqLzZqf8CPEk2wYFV1+ao/V1fVlas+f9wGHufUqnr2Ru1vD489FDxVdWxVvamqLqyqS6rqM1X1pKo6eF/NumqWU6vqu1V12fznn6vqj6vqiJX7dPdru/sBg/ta97nv7rt29wf2cvRU1X2q6mtr9v1H3f2be7tvYPdEEyxYd99o5U+Sf0ly/KrbXrtyv2VaYVmEqrp9kn9Mcm6Sn+ruI5I8KslxSW68SWM9v7tvnGRLkick+bkk/1BVP7aRB9nfv7dwoBBNsElWVguq6qSq+kaSV1fVQVX19Kr6UlVdVFVvrKqbrXrMm6rqG/MqzYeq6q7z7ScmeVySp80rWO+cbz+nqp46r+h8u6peWVW3qKoz5tWV91bVTVft/+eq6qNVtbOqPr1yKmne9oGq+sOq+of5se+uqqPmzR+a/945H/+eu/iSn5Xko939pO7+epJ099ndfUJ379zF8/OEqvrcfKwvV9V/XbXtqKp61zznt6rqw1V10LztpKo6b37c2VV1v/W+F939ne7+P0kenuTITAGVqnp8VX1k/riq6kVVdUFVXVpV/1RVP7nOc39SVX0myber6pD5tvuvOvQNquqv51k/UVV3W/U1dlXdYdXnp1bVs+egOyPJ0atWLI+uNaf7qurhNZ0O3Dl/735i1bZzquop838Xl8wz3GC95wkOdKIJNtctk9wsyTFJTkzy20l+OckvJjk6ycVJXr7q/mckuWOSmyf5RJLXJkl3b5s/fv68gnX8qsf8SpL/kOTYJMfP+3hmptWVg5L8TpJU1a2SnJbk2fNMT0nyN1W1ZdW+TsgUFDdPcr35Pkly7/nvm8zH/9guvtb7J3nz2NOSJLkgycOSHD4f80VV9TPzticn+dr8Ndxi/nq6qu6U5LeS/Oy8gvTAJOeMHrC7L0vyniS/sIvND8j0dR6b5Igkj05y0TrP/a8leWim5+Vfd7HPRyR5U6bn+3VJ3lZVh64z47eTPDjJ+atWLM9ffZ+qOjbJ65P8Xqbn6PQk76yq662626OTPCjJbZP82ySPv6bjAvsomqrqVfO/zv558P6Prqqz5n8lvW7R88EmujrJ73f3Vd19ZZInJvkf3f217r4qyclJfnXl9E53v6q7L1u17W616hqc3Xhpd3+zu89L8uEk/9jdn+zu7yR5a5J/N9/vPyY5vbtP7+6ru/s9SbYneciqfb26u78wz/rGJD+9B1/rkUm+Pnrn7j6tu7/Ukw8meXd+GDPfS/JvkhzT3d/r7g/39Eaa309y/SR3qapDu/uc7v7SHsyYJOdnipi1vpfpNOKdk1R3f25lxewavKS7z52fr105s7vf3N3fS/KnSW6Q6RTh3npMktO6+z3zvv8kyQ2T3GvNbOd397eSvDN79r2EA9K+Wmk6NdO/aNZVVXdM8owk/76775rpX0qwv9oxx8uKY5K8dT6lsjPJ5zKFwC2q6uCqeu586u7S/HAF5ahcs2+u+vjKXXx+o1XHftTKsefj/3ymOFnxjVUfX7HqsSMuWrOva1RVD66qj8+n33ZmireVr/UFSb6Y5N3zqbunJ0l3fzHTz4yTk1xQVW+oqqP3YMYkuVWSb629sbvfn+RlmVb+LqiqbVV1+Dr7Ond0e3dfnWn1bE/n3ZWjk3x1zb7PzfS1rdib7yUckPZJNHX3h7Lmh1BV3b6q/raqzpyvR7jzvOm/JHl5d188P/aCfTEjbJJe8/m5SR7c3TdZ9ecG8yrRCZlO59w/0+mhrfNjajf72lPnJvmrNcf+se5+7rX4OnblvZlOFa6rqq6f5G8yrZDcortvkukUUyXTabTufnJ33y7TdUhPWrl2qbtf190/nykCO8nzRo45H/dGmZ7fD+9qe3e/pLvvnuQumU7TPXVl0252ud7zcutVxz4oyY9nWulKppA5bNV9b7kH+z0/09e/su+aj3XeOo8DrsFmXtO0Lclvzz+AnpLkFfPtxyY5dr7Y9ONVNbRCBfuJU5I8p6qOSZKq2lJVj5i33TjJVZlWbA5L8kdrHvvNJLfbi2O/JsnxVfXAeVXrBjVdrP7jA4/dkelU4zUd//eT3KuqXlBVt0ySqrpDVb2mqm6y5r7Xy3SabUeSf62qB2e6pijz4x42P7aSXJJpNe7qqrpTVf3SHF3fybSSdvV6w1fV9avq7kneluk6slfv4j4/W1X3mK85+va8/5V9X9vn/u5V9cj59OvvZfr+fnze9qkkJ8zfiwdlus5txTeTHHkNp2bfmOShVXW/ed4nz/v+6LWYEZhtSjTN/5q7V5I3VdWnkvxZfrhsf0imC13vk+kiyj/fxQ9U2F+9OMk7Mp12uizT/0DvMW/7y0ynXM5LclZ++D/XFa/MdC3Pzqp6254euLvPzbSS9cxMsXJuppWUdX9OdPcVSZ6T6df1d1bVj1yXM19bdM9MK2SfrapLMq0mbU9y2Zr7XpbpAvU3ZoqYEzI9LyvumGnl6vIkH0vyiu7++0yh9dwkF2Y6/XTzTKf7d+dp8/N8Uabn98wk95ovtl7r8CR/Ps/z1fkxL5i3Xdvn/u2Zrj+6OMl/SvLI+RqkJPndTBfu78z023k/2G93fz7Thd5fno/5/53S6+6zM12j9tJMz8XxmV7q4rt7MBuwRk3XTu6DA1VtTfKu7v7J+TqAs7v7R65vqKpTMl2o+ur58/clefr868AAAJtiU1aauvvSJF+pqkclP3j9k5XXJ3lbplWm1PQaMMcm+fImjAkA8AP76iUHXp9pCf1ONb2Y329kWm7+jar6dJLPZjotkCR/l+Siqjoryd8neWp3X7Qv5gQA2J19dnoOAOC6zCuCAwAMEE0AAAMW/s7bRx11VG/dunXRhwEA2Gtnnnnmhd29ZVfbFh5NW7duzfbt2xd9GACAvVZVX93dNqfnAAAGiCYAgAGiCQBggGgCABggmgAABogmAIABogkAYIBoAgAYIJoAAAaIJgCAAaIJAGCAaAIAGCCaAAAGHLLZAxyobvv00zZ7BK4jvvLch272CADEShMAwBDRBAAwQDQBAAwQTQAAA0QTAMAA0QQAMEA0AQAMEE0AAANEEwDAANEEADBANAEADBBNAAADRBMAwADRBAAwQDQBAAwQTQAAA0QTAMAA0QQAMEA0AQAMEE0AAANEEwDAANEEADBANAEADBBNAAADRBMAwADRBAAwQDQBAAwQTQAAA0QTAMAA0QQAMEA0AQAMEE0AAANEEwDAANEEADBANAEADBBNAAADRBMAwADRBAAwQDQBAAwQTQAAA0QTAMAA0QQAMEA0AQAMGIqmqnpsVX2iqi6vqvOq6i+r6uhFDwcAsCzWjaaqeniS1yf5aJJHJDkpyb2TnFZVVqoAgAPCIQP3OSHJJ7r7t1ZuqKpLk7w9yZ2SfG5BswEALI2RlaJDk1yy5rad89+1odMAACypkWh6VZJfqKr/XFWHV9WxSZ6d5P3dfdZixwMAWA7rRlN3n5bk8Um2ZVpxOjvJwUl+ZXePqaoTq2p7VW3fsWPHBo0KALB5Ri4Ev2+SU5K8OMl9kzw2yc2SvLWqDt7VY7p7W3cf193HbdmyZSPnBQDYFCMXgr8wyTu6+6SVG6rqU0k+n+m36d6ymNEAAJbHyDVNd07yqdU3dPfZSa5McvsFzAQAsHRGoumrSX5m9Q1V9RNJbpjknAXMBACwdEZOz52S5EVVdX6SM5LcIsn/yhRMpy9uNACA5TESTS9J8t0k/y3JEzO9RtNHkjyju7+9uNEAAJbHutHU3Z3kf89/AAAOSN47DgBggGgCABggmgAABogmAIABogkAYIBoAgAYIJoAAAaIJgCAAaIJAGCAaAIAGCCaAAAGiCYAgAGiCQBggGgCABggmgAABogmAIABogkAYIBoAgAYIJoAAAaIJgCAAaIJAGCAaAIAGCCaAAAGiCYAgAGiCQBggGgCABggmgAABogmAIABogkAYIBoAgAYIJoAAAaIJgCAAaIJAGCAaAIAGCCaAAAGiCYAgAGiCQBggGgCABggmgAABogmAIABogkAYIBoAgAYIJoAAAaIJgCAAaIJAGCAaAIAGCCaAAAGiCYAgAGiCQBggGgCABggmgAABogmAIABQ9FUVYdU1dOr6v9W1VVV9bWqetGihwMAWBaHDN7v1CS/lORZST6f5NZJ7rKgmQAAls660VRVD0rymCR36+6zFj8SAMDyGTk99+tJ3i+YAIAD2Ug03SPJF6rqZVV1aVVdUVVvqaqjFz0cAMCyGImmWyZ5fJKfTvLYJE9Icvckb62qWthkAABLZORC8Jr/PKK7L0qSqvp6kg9mujj8fT/ygKoTk5yYJLe5zW02bFgAgM0ystJ0cZJ/Wgmm2UeSfDe7+Q267t7W3cd193FbtmzZgDEBADbXSDR9LtNK01qV5OqNHQcAYDmNRNO7kvxUVR216rZ7Jzk0yacXMhUAwJIZiaZtSS5K8s6qOr6qTkjyV0ne290fWeh0AABLYt1o6u5LM13wfXGSNyR5eaaLvx+92NEAAJbH0NuodPcXkzxkwbMAACytoTfsBQA40IkmAIABogkAYIBoAgAYIJoAAAaIJgCAAaIJAGCAaAIAGCCaAAAGiCYAgAGiCQBggGgCABggmgAABogmAIABogkAYIBoAgAYIJoAAAaIJgCAAaIJAGCAaAIAGCCaAAAGiCYAgAGiCQBggGgCABggmgAABogmAIABogkAYIBoAgAYIJoAAAaIJgCAAaIJAGCAaAIAGCCaAAAGiCYAgAGiCQBggGgCABggmgAABogmAIABogkAYIBoAgAYIJoAAAaIJgCAAaIJAGCAaAIAGCCaAAAGiCYAgAGiCQBggGgCABggmgAABogmAIABogkAYIBoAgAYIJoAAAbscTRV1a2q6vKq6qq60SKGAgBYNtdmpekFSS7f6EEAAJbZHkVTVd07yYOS/MlixgEAWE6HjN6xqg5O8tIkf5Bk56IGAgBYRnuy0vTEJNdP8vIFzQIAsLSGoqmqjkzyh0me1N3fG7j/iVW1vaq279ixY29nBADYdKMrTc9J8vHuPn3kzt29rbuP6+7jtmzZcu2nAwBYEute01RVd03y60nuXVU3mW8+bP77iKr6fndfuaD5AACWwsiF4HdMcmiSj+1i29eSvDLJb27kUAAAy2Ykmj6S5L5rbntQkpOSPCTJlzd6KACAZbNuNHX3hUk+sPq2qto6f/jh7vZClwDAfs97zwEADLhW0dTdp3Z3WWUCAA4UVpoAAAaIJgCAAaIJAGCAaAIAGCCaAAAGiCYAgAGiCQBggGgCABggmgAABogmAIABogkAYIBoAgAYIJoAAAaIJgCAAaIJAGCAaAIAGCCaAAAGiCYAgAGiCQBggGgCABggmgAABogmAIABogkAYIBoAgAYIJoAAAaIJgCAAaIJAGCAaAIAGCCaAAAGiCYAgAGiCQBggGgCABggmgAABogmAIABogkAYIBoAgAYIJoAAAaIJgCAAaIJAGCAaAIAGCCaAAAGiCYAgAGiCQBggGgCABggmgAABogmAIABogkAYIBoAgAYIJoAAAaIJgCAAaIJAGCAaAIAGLBuNFXVo6rqHVV1XlVdXlVnVtWv7YvhAACWxSED93lSkq8k+e9JLkzykCSvq6qjuvulixwOAGBZjETT8d194arP319VR2eKKdEEABwQ1j09tyaYVnwyydEbPw4AwHK6theC3zPJFzZyEACAZbbH0VRV90vyy0leeA33ObGqtlfV9h07duzFeAAAy2GPoqmqtiZ5XZK3d/epu7tfd2/r7uO6+7gtW7bs3YQAAEtgOJqq6mZJzkjy1SSPW9hEAABLaCiaquqwJO9Kcr0kD+vuKxY6FQDAkln3JQeq6pAkb0pyxyT36u4LFj4VAMCSGXmdpldkekHL301yZFUduWrbJ7v7qoVMBgCwREai6QHz3y/exbbbJjlnw6YBAFhS60ZTd2/dB3MAACy1a/vilgAABxTRBAAwQDQBAAwYuRAcgOuKk4/Y7Am4rjj5ks2e4DrHShMAwADRBAAwQDQBAAwQTQAAA0QTAMAA0QQAMEA0AQAMEE0AAANEEwDAANEEADBANAEADBBNAAADRBMAwADRBAAwQDQBAAwQTQAAA0QTAMAA0QQAMEA0AQAMEE0AAANEEwDAANEEADBANAEADBBNAAADRBMAwADRBAAwQDQBAAwQTQAAA0QTAMAA0QQAMEA0AQAMEE0AAANEEwDAANEEADBANAEADBBNAAADRBMAwADRBAAwQDQBAAwQTQAAA0QTAMAA0QQAMEA0AQAMEE0AAANEEwDAANEEADBANAEADBBNAAADhqKpqu5SVe+rqiuq6vyq+oOqOnjRwwEALItD1rtDVd00yXuTnJXkEUlun+SFmYLrfy50OgCAJbFuNCV5YpIbJnlkd1+a5D1VdXiSk6vq+fNtAAD7tZHTcw9O8ndr4ugNmULqFxcyFQDAkhmJpjsn+fzqG7r7X5JcMW8DANjvjUTTTZPs3MXtF8/bAAD2eyPXNO2xqjoxyYnzp5dX1dmLOA77naOSXLjZQyybet5mTwDXeX627MqzarMnWFbH7G7DSDRdnOSIXdx+03nbj+jubUm2DY0Gs6ra3t3HbfYcwP7FzxY2ysjpuc9nzbVLVXXrJIdlzbVOAAD7q5FoOiPJA6vqxqtue0ySK5N8cCFTAQAsmZFoOiXJVUneUlX3n69XOjnJn3qNJjaYU7rAIvjZwoao7l7/TlV3SfKyJPfM9Jt0f5Hk5O7+/kKnAwBYEkPRBABwoBt6w15YlKq6Q1X9WVV9pqq+X1Uf2OyZgP2DN5tnoy3kdZpgD9w1yUOSfDzJoZs8C7Cf8GbzLILTc2yqqjqou6+eP35zkqO6+z6bOxVwXVdVz0jytCTHrPzSUlU9LdMvMt3SLzJxbTg9x6ZaCSaADebN5tlwogmA/ZE3m2fDiSYA9kfebJ4NJ5oAAAaIJgD2R3v8ZvOwHtEEwP7Im82z4UQTAPsjbzbPhvPilmyqqjos04tbJsmtkhxeVb86f356d1+xOZMB13GnJPmdTG82/7wkt4s3m2cveXFLNlVVbU3yld1svm13n7PvpgH2J95sno0mmgAABrimCQBggGgCABggmgAABogmAIABogkAYIBoAgAYIJoAAAaIJgCAAaIJAGDA/wMVGdfHu8yOMAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "print_proportion(train_df, treatment_column)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Random Undersampling (finding the majority class and undersampling it)\n",
    "# https://www.kaggle.com/code/hughhuyton/criteo-uplift-modelling/notebook\n",
    "def random_under(df, feature):\n",
    "    \n",
    "    target = df[feature].value_counts()\n",
    "    \n",
    "    if target.values[0]<target.values[1]:\n",
    "        under = target.index.values[1]\n",
    "    \n",
    "    else: \n",
    "        under = target.index.values[0]\n",
    "        \n",
    "    df_0 = df[df[feature] != under]\n",
    "    df_1 = df[df[feature] == under]\n",
    "    \n",
    "    df_treatment_under = df_1.sample(len(df_0))\n",
    "    df_1 = pd.concat([df_treatment_under, df_0], axis=0)\n",
    "    \n",
    "    return df_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = random_under(train_df, treatment_column)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Class 0: 1677550\n",
      "Class 1: 1677550\n",
      "Proportion: 1 : 1\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlsAAAF6CAYAAAAu3Z1/AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAfNElEQVR4nO3dfZRlVX3m8e8DjfIiNB0pJCjSiEEGHDMztktxzOALiqAEgy9kjFmjgekxa6mT+K5xLVuMjugImYgZQAlGgzJgFIPyIpCA7yRNjCZiEzG2IiTaYDWIjQbp3/xxToWborrrVHXvrrrV389ad1Xdvc8+53dvsYqnz953V6oKSZIktbHLQhcgSZK0lBm2JEmSGjJsSZIkNWTYkiRJasiwJUmS1JBhS5IkqSHDlqSxlmRlkkqybKFrmZLkN5J8djue7xtJntp/vybJn27Hc785yQe31/kkPZBhS1qkktw98tic5J6R57+xHa/zoSS/v73ON8drDwpKSQ5LcnGS25PcmeTrSV6dZNcdVetILR9K8i9Jftw//j7J/0qyfOqYqrqgqp418FyzvvdVdWRVXbuNpZPkqUm+P+3c76yqU7f13JK2zLAlLVJV9ZCpB/A94ISRtgumjltMd3RaSHIocD1wC/Dvq2o58EJgFbD3ApX17qraG5gAXgY8Cfhikr2250WW+s9W2lkYtqQxM3V3IskbkvwzcH6SXZK8Mcm3k9yR5KIkvzAy5uIk/9zfFfpckiP79tXAbwCv7++YXdq3r0/yuv4O0k+SnJfkYUku7+/mXJ1kxcj5n5TkS0k2Jvna1JRX33dtkrcn+WI/9rNJ9uu7P9d/3dhf/6gZXvLbgC9V1aur6p8AquqmqnpxVW2c4f15WZJv9tf6xyT/Y6RvvySf7uv8UZLPJ9ml73tDklv7cTclecZsP4uq+mlV/TXwq8BD6YIXSV6a5Av990lyZpIfJrkryd8leews7/0bknwd+EmSZX3bMSOX3j3J/+tr/ZskvzzyGivJo0eefyjJ7/dB8HLgwJE7pAdm2rRkkl9NN225sf/Z/buRvvVJXtv/d3FnX8Pus71P0s7OsCWNpwOAXwAOBlYDrwSeBxwNHAhMAu8fOf5y4JeA/YG/AS4AqKpz++/f3d8xO2FkzPOBZwKHASf053gz3d2cXYBXASR5OPAZ4Pf7ml4L/FmSiZFzvZguiOwPPKg/BuC/9F/37a//5Rle6zHAx4e9LQD8EHgusE9/zTOT/Ke+7zXA9/vX8LD+9VSSxwCvAJ7Q37E6Flg/9IJV9WPgKuBXZuh+Ft3rPAxYDrwIuGOW9/6/As+he19+PsM5TwQupnu/PwpckmS3WWr8CXAccNvIHdLbRo9JchjwMeB36N6jy4BLkzxo5LAXAc8GDgEeB7x0a9eVtMjDVpI/7v81+PcDj39Rkhv7f5V9tHV90gLaDLy1qn5WVfcALwd+r6q+X1U/A9YAL5iahqqqP66qH4/0/XJG1hhtwfuq6gdVdSvweeD6qvpqVf0U+CTwH/vjXgJcVlWXVdXmqroKWAscP3Ku86vqH/paLwL+wxxe60OBfxp6cFV9pqq+XZ3rgM9yfwi6F/hF4OCqureqPl/dH4i9D3gwcESS3apqfVV9ew41AtxGF36mu5duuvNwIFX1zak7dFvxh1V1S/9+zeSGqvp4Vd0LnAHsTjeVua1OBj5TVVf15/7fwB7Ak6fVdltV/Qi4lLn9LKWd0qIOW8CH6P4FNaskvwS8CfjPVXUk3b/MpKVqQx96phwMfLKf+tkIfJMuQDwsya5J3tVPMd7F/Xds9mPrfjDy/T0zPH/IyLVfOHXt/vpPoQs1U/555PtNI2OHuGPaubYqyXFJvtJPE26kC31Tr/U9wM3AZ/spxjcCVNXNdL8z1gA/THJhkgPnUCPAw4EfTW+sqr8AzqK70/jDJOcm2WeWc90ytL+qNtPdrZtrvTM5EPjutHPfQvfapmzLz1LaKS3qsFVVn2PaL68khya5IskN/XqLw/uu/w68v6om+7E/3MHlSjtSTXt+C3BcVe078ti9vyv1Yrppp2PoprFW9mOyhXPN1S3AR6Zde6+qetc8XsdMrqab0pxVkgcDf0Z3R+ZhVbUv3VRYoJvuq6rXVNWj6NZZvXpqbVZVfbSqnkIXHgs4fcg1++s+hO79/fxM/VX1h1X1eOAIuunE1011beGUs70vB41cexfgEXR31qALQHuOHHvAHM57G93rnzp3+mvdOss4SVuxqMPWFpwLvLL/xfVa4I/69sOAw/pFuF9JMuiOmLREnA28I8nBAEkmkpzY9+0N/IzuDtGewDunjf0B8KhtuPafAickOba/i7Z7ukX8jxgwdgPdlOjWrv9W4MlJ3pPkAIAkj07yp0n2nXbsg+imAzcAP09yHN2aKfpxz+3HBriT7u7f5iSPSfL0Pqz9lO7O3ebZik/y4CSPBy6hWyd3/gzHPCHJE/s1VT/pzz917vm+949PclI/Tfw7dD/fr/R9fwu8uP9ZPJtuHd+UHwAP3coU8kXAc5I8o6/3Nf25vzSPGiX1xips9f96fDJwcZK/Bc7h/umFZXQLgJ9Kt7j0AzP8IpaWqv8D/Dnd9NiP6f7H+8S+78N0U0O3Ajdy//+Up5xHt1ZpY5JL5nrhqrqF7s7Zm+lCzi10d25m/f1SVZuAd9Btm7AxyQPWHfVrp46iuyP3jSR30t29Wgv8eNqxP6ZbuH8RXfh5Md37MuWX6O6U3Q18GfijqvpLuoD2LuB2ummy/emWJWzJ6/v3+Q669/cG4Mn9IvTp9gE+0Nfz3X7Me/q++b73n6JbXzUJ/CZwUr/GCuB/0n2gYSPdpx3/9bxVtY5uAfw/9tf8N1OPVXUT3Rq899G9FyfQbTnyL3OoTdI06daGLl5JVgKfrqrH9uscbqqqB6zfSHI23QLe8/vn1wBv7D+WLUmStCDG6s5WVd0FfCfJC+Ff96+Z2l/mErq7WqTbw+cw4B8XoExJkqR/tajDVpKP0d3qf0y6TRxPobstfkqSrwHfoJu+ALgSuCPJjcBfAq+rqjsWom5JkqQpi34aUZIkaZwt6jtbkiRJ486wJUmS1NCi/Yvy++23X61cuXKhy5AkSZrVDTfccHtVTczUt2jD1sqVK1m7du1ClyFJkjSrJN/dUp/TiJIkSQ0ZtiRJkhoybEmSJDVk2JIkSWrIsCVJktSQYUuSJKkhw5YkSVJDhi1JkqSGDFuSJEkNGbYkSZIaMmxJkiQ1ZNiSJElqyLAlSZLU0LKFLkBzc8gbP7PQJWhMfOddz1noEjRO1ixf6Ao0LtbcudAVjB3vbEmSJDVk2JIkSWrIsCVJktSQYUuSJKkhw5YkSVJDhi1JkqSGDFuSJEkNGbYkSZIaGhS2kjw6yTlJvp7kviTXDr1AkpOS/HWSe5LckeSKJHvNu2JJkqQxMvTO1pHA8cBNwD8MPXmSU4GPApcDxwGnAt/CneslSdJOYmjoubSqPgWQ5OPAfrMNSLIfcCbwyqr6wEjXJ+dcpSRJ0pgadGerqjbP49wv6r/+yTzGSpIkLQktF8g/kW7a8ZQk309yb5Lrkzy54TUlSZIWlZZh6wDgMcBbgDcAJwA/Aa5I8rCG15UkSVo0WoatAA8BTqmqC6rqCuB5wH3AK2YckKxOsjbJ2g0bNjQsTZIkacdoGbYmgQKunWqoqruAG4AjZhpQVedW1aqqWjUxMdGwNEmSpB2jZdj6Jt3drUxrDzCfBfeSJEljp2XY+nT/9WlTDUmWA48HvtbwupIkSYvGoH22kuxJt6kpwMOBfZK8oH9+WVVtSnIzcF1VnQJQVWuTfAo4L8kbgduB1wP3Au/fni9CkiRpsRq6qen+wMXT2qaeHwKs78+167RjXgK8BzgD2BP4IvD0qpqcT7GSJEnjZlDYqqr1PHDt1fRjVs7Qdjfw2/1DkiRpp9NyzZYkSdJOz7AlSZLUkGFLkiSpIcOWJElSQ4YtSZKkhgxbkiRJDRm2JEmSGjJsSZIkNWTYkiRJasiwJUmS1JBhS5IkqSHDliRJUkOGLUmSpIYMW5IkSQ0ZtiRJkhoybEmSJDVk2JIkSWrIsCVJktSQYUuSJKkhw5YkSVJDhi1JkqSGDFuSJEkNGbYkSZIaMmxJkiQ1NChsJXl0knOSfD3JfUmunctFkuySZG2SSvLceVUqSZI0hpYNPO5I4HjgK8Bu87jOqcAj5jFOkiRprA2dRry0qg6qqhcC35jLBZKsAN4B/N5ci5MkSRp3g8JWVW3ehmu8HfgicM02nEOSJGksDZ1GnJckjwN+C3hcy+tIkiQtVq0/jfg+4KyqurnxdSRJkhalZne2kvw68BjghDmMWQ2sBnjkIx/ZqDJJkqQdp8mdrSS7Ae8BTgd2SbIvsE/fvVeSvWcaV1XnVtWqqlo1MTHRojRJkqQdqtU04l50Wz2cAUz2j6/1fRcCX210XUmSpEWl1TTi3cDTprUdAHwMeDPwF42uK0mStKgMCltJ9qTb1BTg4cA+SV7QP7+sqjYluRm4rqpOqaqfA9dOO8fK/tu/q6rrt7lySZKkMTD0ztb+wMXT2qaeHwKs78+16/YpS5IkaWkYFLaqaj2QWY5Zua3nkCRJWmpa77MlSZK0UzNsSZIkNWTYkiRJasiwJUmS1JBhS5IkqSHDliRJUkOGLUmSpIYMW5IkSQ0ZtiRJkhoybEmSJDVk2JIkSWrIsCVJktSQYUuSJKkhw5YkSVJDhi1JkqSGDFuSJEkNGbYkSZIaMmxJkiQ1ZNiSJElqyLAlSZLUkGFLkiSpIcOWJElSQ4YtSZKkhgxbkiRJDQ0KW0keneScJF9Pcl+SaweMeUKS85PcnGRTkpuSvDXJ7ttctSRJ0phYNvC4I4Hjga8Auw0cczJwKHA68C3gccDb+6/Pn1uZkiRJ42lo2Lq0qj4FkOTjwH4Dxryrqm4feX5tkp8C5yQ5uKq+O8daJUmSxs6gacSq2jzXE08LWlO+2n89cK7nkyRJGkc7eoH8UcBm4Ns7+LqSJEkLYoeFrSQHAG8BPlJVP9xR15UkSVpIOyRsJXkQcBFwN/C7WzludZK1SdZu2LBhR5QmSZLUVPOwlSTAh+k/0VhVk1s6tqrOrapVVbVqYmKidWmSJEnNDf004rb4A+BE4JlVtW4HXE+SJGnRaBq2krwJeAXwoqr6QstrSZIkLUaDwlaSPek2NQV4OLBPkhf0zy+rqk1Jbgauq6pT+jEvBt4JfAi4NcmTRk757apyUZYkSVryht7Z2h+4eFrb1PNDgPX9uXYd6X9W//Wl/WPUy+hCmCRJ0pI2KGxV1Xogsxyzctrzl/LAkCVJkrRT2dGbmkqSJO1UDFuSJEkNGbYkSZIaMmxJkiQ1ZNiSJElqyLAlSZLUkGFLkiSpIcOWJElSQ4YtSZKkhgxbkiRJDRm2JEmSGjJsSZIkNWTYkiRJasiwJUmS1JBhS5IkqSHDliRJUkOGLUmSpIYMW5IkSQ0ZtiRJkhoybEmSJDVk2JIkSWrIsCVJktSQYUuSJKkhw5YkSVJDg8JWkkcnOSfJ15Pcl+TageOWJzk/yWSSO5NckOSh21SxJEnSGFk28LgjgeOBrwC7zeH8FwGHAacCm4HTgUuAX5nDOSRJksbW0LB1aVV9CiDJx4H9ZhuQ5CjgWcDRVfW5vu1W4Pokx1TV1fOsWZIkaWwMmkasqs3zOPdxwA+mglZ/nr8CvtP3SZIkLXktF8gfDqybof2bfZ8kSdKS1zJsrQA2ztA+2fdJkiQteYtq64ckq5OsTbJ2w4YNC12OJEnSNmsZtiaB5TO0r+j7HqCqzq2qVVW1amJiomFpkiRJO0bLsLWOmddmbWktlyRJ0pLTMmxdDhyQ5ClTDUlWAY/q+yRJkpa8QftsJdmTblNTgIcD+yR5Qf/8sqralORm4LqqOgWgqr6c5LPAh5O8lvs3Nf2Ce2xJkqSdxdBNTfcHLp7WNvX8EGB9f65dpx1zMnAm8Md0d9E+DbxqPoVKkiSNo0Fhq6rWA5nlmJUztG0EXtY/JEmSdjqLausHSZKkpcawJUmS1JBhS5IkqSHDliRJUkOGLUmSpIYMW5IkSQ0ZtiRJkhoybEmSJDVk2JIkSWrIsCVJktSQYUuSJKkhw5YkSVJDhi1JkqSGDFuSJEkNGbYkSZIaMmxJkiQ1ZNiSJElqyLAlSZLUkGFLkiSpIcOWJElSQ4YtSZKkhgxbkiRJDRm2JEmSGjJsSZIkNTQobCU5Isk1STYluS3JaUl2HTBuVZLPJvlR/7g6yRO3vWxJkqTxMGvYSrICuBoo4ETgNOA1wNtmGXdQP24Z8Jv9YxlwVZKDt61sSZKk8bBswDEvB/YATqqqu+jC0j7AmiTv7ttm8hxgb+DXqupOgCRfAm4Hjgf+7zZXL0mStMgNmUY8DrhyWqi6kC6AHb2VcbsBPwd+MtJ2d9+WOdYpSZI0loaErcOBdaMNVfU9YFPftyV/1h/z3iT7J9kfOBOYBC6eX7mSJEnjZUjYWgFsnKF9su+bUVXdBjwNeD7wg/5xEnBsVW2Yc6WSJEljqNnWD0l+ke4O1g10U5HH9d9/JskjtzBmdZK1SdZu2GAekyRJ429I2JoEls/QvqLv25LX0a3bekFVXVFVV9Dd5boPeO1MA6rq3KpaVVWrJiYmBpQmSZK0uA0JW+uYtjar39ZhT6at5ZrmcOAbVXXvVENV/QvwDeDQuZcqSZI0foaErcuBY5PsPdJ2MnAPcN1Wxn0XeGySB001JHkw8Fhg/dxLlSRJGj9DwtbZwM+ATyQ5JslqYA1wxuh2EEluTnLeyLgPAgcCn0zynCTPBS4BfhE4dzvVL0mStKjNGraqahJ4BrArcCndzvFnAm+dduiy/pipcTcAz6bb2PQjwIfpph6fWVVf2x7FS5IkLXZDdpCnqm4Enj7LMStnaLsGuGZelUmSJC0BzbZ+kCRJkmFLkiSpKcOWJElSQ4YtSZKkhgxbkiRJDRm2JEmSGjJsSZIkNWTYkiRJasiwJUmS1JBhS5IkqSHDliRJUkOGLUmSpIYMW5IkSQ0ZtiRJkhoybEmSJDVk2JIkSWrIsCVJktSQYUuSJKkhw5YkSVJDhi1JkqSGDFuSJEkNGbYkSZIaMmxJkiQ1ZNiSJElqaFDYSnJEkmuSbEpyW5LTkuw6cOxJSf46yT1J7khyRZK9tq1sSZKk8TBr2EqyArgaKOBE4DTgNcDbBow9FfgocDlwHHAq8C1g2fxLliRJGh9DQs/LgT2Ak6rqLuCqJPsAa5K8u297gCT7AWcCr6yqD4x0fXJbi5YkSRoXQ6YRjwOunBaqLqQLYEdvZdyL+q9/Ms/aJEmSxt6QsHU4sG60oaq+B2zq+7bkicBNwClJvp/k3iTXJ3nyvKuVJEkaM0PC1gpg4wztk33flhwAPAZ4C/AG4ATgJ8AVSR42tzIlSZLGU8utHwI8BDilqi6oqiuA5wH3Aa+YcUCyOsnaJGs3bNjQsDRJkqQdY0jYmgSWz9C+ou/b2rgCrp1q6Nd93QAcMdOAqjq3qlZV1aqJiYkBpUmSJC1uQ8LWOqatzUpyELAn09ZyTfNNurtbmdYeYPMcapQkSRpbQ8LW5cCxSfYeaTsZuAe4bivjPt1/fdpUQ5LlwOOBr82xTkmSpLE0JGydDfwM+ESSY5KsBtYAZ4xuB5Hk5iTnTT2vqrXAp4Dzkvy3JM8B/hy4F3j/dnwNkiRJi9asYauqJoFnALsCl9LtHH8m8NZphy7rjxn1EuAS4Azg43RB6+n9OSVJkpa8QX82p6puBJ4+yzErZ2i7G/jt/iFJkrTTabn1gyRJ0k7PsCVJktSQYUuSJKkhw5YkSVJDhi1JkqSGDFuSJEkNGbYkSZIaMmxJkiQ1ZNiSJElqyLAlSZLUkGFLkiSpIcOWJElSQ4YtSZKkhgxbkiRJDRm2JEmSGjJsSZIkNWTYkiRJasiwJUmS1JBhS5IkqSHDliRJUkOGLUmSpIYMW5IkSQ0ZtiRJkhoybEmSJDU0KGwlOSLJNUk2JbktyWlJdh16kSS7JFmbpJI8d/7lSpIkjZdlsx2QZAVwNXAjcCJwKPBeuqD2loHXORV4xDxrlCRJGltD7my9HNgDOKmqrqqqs4G3Aa9Oss9sg/uw9g7g97apUkmSpDE0JGwdB1xZVXeNtF1IF8COHjD+7cAXgWvmXp4kSdJ4GxK2DgfWjTZU1feATX3fFiV5HPBbwGvnW6AkSdI4GxK2VgAbZ2if7Pu25n3AWVV18xzrkiRJWhJmXSA/X0l+HXgMcMIcxqwGVgM88pGPbFSZJEnSjjPkztYksHyG9hV93wMk2Q14D3A6sEuSfYGpxfR7Jdl7pnFVdW5VraqqVRMTEwNKkyRJWtyGhK11TFubleQgYE+mreUasRfdVg9n0AWySeBrfd+FwFfnU6wkSdK4GTKNeDnwuiR7V9WP+7aTgXuA67Yw5m7gadPaDgA+BrwZ+It51CpJkjR2hoSts4FXAZ9IcjrwKGANcMbodhBJbgauq6pTqurnwLWjJ0mysv/276rq+m0vXZIkafGbNWxV1WSSZwBnAZfSfTLxTLrANf1cg/+EjyRJ0s5g0KcRq+pG4OmzHLNylv71QIYWJkmStBQM+kPUkiRJmh/DliRJUkOGLUmSpIYMW5IkSQ0ZtiRJkhoybEmSJDVk2JIkSWrIsCVJktSQYUuSJKkhw5YkSVJDhi1JkqSGDFuSJEkNGbYkSZIaMmxJkiQ1ZNiSJElqyLAlSZLUkGFLkiSpIcOWJElSQ4YtSZKkhgxbkiRJDRm2JEmSGjJsSZIkNWTYkiRJasiwJUmS1NCgsJXkiCTXJNmU5LYkpyXZdZYxT0hyfpKb+3E3JXlrkt23T+mSJEmL37LZDkiyArgauBE4ETgUeC9dUHvLVoae3B97OvAt4HHA2/uvz9+mqiVJksbErGELeDmwB3BSVd0FXJVkH2BNknf3bTN5V1XdPvL82iQ/Bc5JcnBVfXfbSpckSVr8hkwjHgdcOS1UXUgXwI7e0qBpQWvKV/uvBw6uUJIkaYwNCVuHA+tGG6rqe8Cmvm8ujgI2A9+e4zhJkqSxNCRsrQA2ztA+2fcNkuQAujVeH6mqHw4dJ0mSNM52yNYPSR4EXATcDfzuVo5bnWRtkrUbNmzYEaVJkiQ1NSRsTQLLZ2hf0fdtVZIAHwaOBI6vqi2Oqapzq2pVVa2amJgYUJokSdLiNuTTiOuYtjYryUHAnkxby7UFf0C3ZcQzq2rI8ZIkSUvGkDtblwPHJtl7pO1k4B7guq0NTPIm4BXAS6rqC/OuUpIkaUwNCVtnAz8DPpHkmCSrgTXAGaPbQfQ7xZ838vzFwDvpphBvTfKkkYdzhJIkaacw6zRiVU0meQZwFnAp3ScTz6QLXNPPNfonfJ7Vf31p/xj1MuBDc6xVkiRp7AxZs0VV3Qg8fZZjVk57/lIeGLIkSZJ2Kjtk6wdJkqSdlWFLkiSpIcOWJElSQ4YtSZKkhgxbkiRJDRm2JEmSGjJsSZIkNWTYkiRJasiwJUmS1JBhS5IkqSHDliRJUkOGLUmSpIYMW5IkSQ0ZtiRJkhoybEmSJDVk2JIkSWrIsCVJktSQYUuSJKkhw5YkSVJDhi1JkqSGDFuSJEkNGbYkSZIaMmxJkiQ1ZNiSJElqaFDYSnJEkmuSbEpyW5LTkuw6YNzyJOcnmUxyZ5ILkjx028uWJEkaD8tmOyDJCuBq4EbgROBQ4L10Qe0tswy/CDgMOBXYDJwOXAL8yrwrliRJGiOzhi3g5cAewElVdRdwVZJ9gDVJ3t23PUCSo4BnAUdX1ef6tluB65McU1VXb5+XIEmStHgNmUY8DrhyWqi6kC6AHT3LuB9MBS2Aqvor4Dt9nyRJ0pI3JGwdDqwbbaiq7wGb+r7B43rfnGWcJEnSkjEkbK0ANs7QPtn3be9xkiRJS8aQNVs7TJLVwOr+6d1JblrIejRW9gNuX+giFpOcvtAVSEuCv1ume1sWuoLF6uAtdQwJW5PA8hnaV/R9Wxs3MZdxVXUucO6AmqR/I8naqlq10HVIWlr83aLtYcg04jqmrbFKchCwJzOvydriuN6W1nJJkiQtOUPC1uXAsUn2Hmk7GbgHuG6WcQckecpUQ5JVwKP6PkmSpCVvSNg6G/gZ8Ikkx/TrqtYAZ4xuB5Hk5iTnTT2vqi8DnwU+nOSkJM8DLgC+4B5basDpZ0kt+LtF2yxVNftByRHAWcBRdJ8w/CCwpqruGzlmPXBtVb10pG1f4Ezg1+iC3aeBV1WViw0lSdJOYVDYkiRJ0vwM+kPU0mKU5NFJzkny9ST3Jbl2oWuSNP6SHJHkmiSbktyW5LQkuy50XRpfi2qfLWmOjgSOB74C7LbAtUhaApKsAK4GbgROBA4F3kt3c+ItC1iaxpjTiBpbSXapqs399x8H9quqpy5sVZLGWZI3Aa8HDp76EFiS19N9MOyAaX8nWBrEaUSNramgJUnb0XHAldNC1YXAHsDRC1OSxp1hS5Kk+z1g4+2q+h6wiZk36pZmZdiSJOl+K+i2OJpusu+T5sywJUmS1JBhS5Kk+00Cy2doX9H3SXNm2JIk6X7rmLY2K8lBwJ5MW8slDWXYkiTpfpcDxybZe6TtZOAe4LqFKUnjzk1NNbaS7Em3qSnAw4F9krygf35ZVW1amMokjbGzgVcBn0hyOvAouj22znCPLc2Xm5pqbCVZCXxnC92HVNX6HVeNpKUiyRHAWcBRdJ9M/CCwpqruW8i6NL4MW5IkSQ25ZkuSJKkhw5YkSVJDhi1JkqSGDFuSJEkNGbYkSZIaMmxJkiQ1ZNiSJElqyLAlSZLUkGFLkiSpof8PNkOpgYt6UW8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "print_proportion(train_df, treatment_column)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# method to transfor data for LGWUM method, will be explained later\n",
    "def target_class_lgwum(df, treatment, target, column_name):\n",
    "    \n",
    "    #CN:\n",
    "    df[column_name] = 0 \n",
    "    #CR:\n",
    "    df.loc[(df[treatment] == 0) & (df[target] != 0), column_name] = 1 \n",
    "    #TN:\n",
    "    df.loc[(df[treatment] != 0) & (df[target] == 0), column_name] = 2 \n",
    "    #TR:\n",
    "    df.loc[(df[treatment] != 0) & (df[target] != 0), column_name] = 3 \n",
    "    return df\n",
    "\n",
    "response_column_lgwum = \"lqwum_response\"\n",
    "train_df = target_class_lgwum(train_df, treatment_column, response_column, response_column_lgwum)\n",
    "test_df = target_class_lgwum(test_df, treatment_column, response_column, response_column_lgwum)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Start H2O"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking whether there is an H2O instance running at http://localhost:54321. connected.\n",
      "Warning: Version mismatch. H2O is version 3.42.0.99999, but the h2o-python package is version 0.0.local. This is a developer build, please contact your developer.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "\n",
       "#h2o-table-1.h2o-container {\n",
       "  overflow-x: auto;\n",
       "}\n",
       "#h2o-table-1 .h2o-table {\n",
       "  /* width: 100%; */\n",
       "  margin-top: 1em;\n",
       "  margin-bottom: 1em;\n",
       "}\n",
       "#h2o-table-1 .h2o-table caption {\n",
       "  white-space: nowrap;\n",
       "  caption-side: top;\n",
       "  text-align: left;\n",
       "  /* margin-left: 1em; */\n",
       "  margin: 0;\n",
       "  font-size: larger;\n",
       "}\n",
       "#h2o-table-1 .h2o-table thead {\n",
       "  white-space: nowrap; \n",
       "  position: sticky;\n",
       "  top: 0;\n",
       "  box-shadow: 0 -1px inset;\n",
       "}\n",
       "#h2o-table-1 .h2o-table tbody {\n",
       "  overflow: auto;\n",
       "}\n",
       "#h2o-table-1 .h2o-table th,\n",
       "#h2o-table-1 .h2o-table td {\n",
       "  text-align: right;\n",
       "  /* border: 1px solid; */\n",
       "}\n",
       "#h2o-table-1 .h2o-table tr:nth-child(even) {\n",
       "  /* background: #F5F5F5 */\n",
       "}\n",
       "\n",
       "</style>      \n",
       "<div id=\"h2o-table-1\" class=\"h2o-container\">\n",
       "  <table class=\"h2o-table\">\n",
       "    <caption></caption>\n",
       "    <thead></thead>\n",
       "    <tbody><tr><td>H2O_cluster_uptime:</td>\n",
       "<td>04 secs</td></tr>\n",
       "<tr><td>H2O_cluster_timezone:</td>\n",
       "<td>Europe/Berlin</td></tr>\n",
       "<tr><td>H2O_data_parsing_timezone:</td>\n",
       "<td>UTC</td></tr>\n",
       "<tr><td>H2O_cluster_version:</td>\n",
       "<td>3.42.0.99999</td></tr>\n",
       "<tr><td>H2O_cluster_version_age:</td>\n",
       "<td>15 minutes</td></tr>\n",
       "<tr><td>H2O_cluster_name:</td>\n",
       "<td>mori</td></tr>\n",
       "<tr><td>H2O_cluster_total_nodes:</td>\n",
       "<td>1</td></tr>\n",
       "<tr><td>H2O_cluster_free_memory:</td>\n",
       "<td>5.113 Gb</td></tr>\n",
       "<tr><td>H2O_cluster_total_cores:</td>\n",
       "<td>8</td></tr>\n",
       "<tr><td>H2O_cluster_allowed_cores:</td>\n",
       "<td>8</td></tr>\n",
       "<tr><td>H2O_cluster_status:</td>\n",
       "<td>locked, healthy</td></tr>\n",
       "<tr><td>H2O_connection_url:</td>\n",
       "<td>http://localhost:54321</td></tr>\n",
       "<tr><td>H2O_connection_proxy:</td>\n",
       "<td>{\"http\": null, \"https\": null}</td></tr>\n",
       "<tr><td>H2O_internal_security:</td>\n",
       "<td>False</td></tr>\n",
       "<tr><td>Python_version:</td>\n",
       "<td>3.7.3 candidate</td></tr></tbody>\n",
       "  </table>\n",
       "</div>\n"
      ],
      "text/plain": [
       "--------------------------  -----------------------------\n",
       "H2O_cluster_uptime:         04 secs\n",
       "H2O_cluster_timezone:       Europe/Berlin\n",
       "H2O_data_parsing_timezone:  UTC\n",
       "H2O_cluster_version:        3.42.0.99999\n",
       "H2O_cluster_version_age:    15 minutes\n",
       "H2O_cluster_name:           mori\n",
       "H2O_cluster_total_nodes:    1\n",
       "H2O_cluster_free_memory:    5.113 Gb\n",
       "H2O_cluster_total_cores:    8\n",
       "H2O_cluster_allowed_cores:  8\n",
       "H2O_cluster_status:         locked, healthy\n",
       "H2O_connection_url:         http://localhost:54321\n",
       "H2O_connection_proxy:       {\"http\": null, \"https\": null}\n",
       "H2O_internal_security:      False\n",
       "Python_version:             3.7.3 candidate\n",
       "--------------------------  -----------------------------"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "h2o.init(strict_version_check=False) # max_mem_size=10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import data to H2O"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parse progress: |████████████████████████████████████████████████████████████████| (done) 100%\n",
      "Parse progress: |████████████████████████████████████████████████████████████████| (done) 100%\n"
     ]
    }
   ],
   "source": [
    "h2o_train_df = h2o.H2OFrame(train_df)\n",
    "del(train_df)\n",
    "\n",
    "h2o_train_df[treatment_column] = h2o_train_df[treatment_column].asfactor()\n",
    "h2o_train_df[response_column] = h2o_train_df[response_column].asfactor()\n",
    "h2o_train_df[response_column_lgwum] = h2o_train_df[response_column_lgwum].asfactor()\n",
    "h2o_train_df = h2o.assign(h2o_train_df, \"train_df\")\n",
    "\n",
    "h2o_test_df = h2o.H2OFrame(test_df)\n",
    "del(test_df)\n",
    "h2o_test_df[treatment_column] = h2o_test_df[treatment_column].asfactor()\n",
    "h2o_test_df[response_column] = h2o_test_df[response_column].asfactor()\n",
    "h2o_test_df[response_column_lgwum] = h2o_test_df[response_column_lgwum].asfactor()\n",
    "h2o_test_df = h2o.assign(h2o_test_df, \"test_df\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train H2O UpliftDRF model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "upliftdrf Model Build progress: |████████████████████████████████████████████████| (done) 100%\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style='margin: 1em 0 1em 0;'>Model Details\n",
       "=============\n",
       "H2OUpliftRandomForestEstimator : Uplift Distributed Random Forest\n",
       "Model Key: UpliftDRF_model_python_1689859822235_1\n",
       "</pre>\n",
       "<div style='margin: 1em 0 1em 0;'>\n",
       "<style>\n",
       "\n",
       "#h2o-table-2.h2o-container {\n",
       "  overflow-x: auto;\n",
       "}\n",
       "#h2o-table-2 .h2o-table {\n",
       "  /* width: 100%; */\n",
       "  margin-top: 1em;\n",
       "  margin-bottom: 1em;\n",
       "}\n",
       "#h2o-table-2 .h2o-table caption {\n",
       "  white-space: nowrap;\n",
       "  caption-side: top;\n",
       "  text-align: left;\n",
       "  /* margin-left: 1em; */\n",
       "  margin: 0;\n",
       "  font-size: larger;\n",
       "}\n",
       "#h2o-table-2 .h2o-table thead {\n",
       "  white-space: nowrap; \n",
       "  position: sticky;\n",
       "  top: 0;\n",
       "  box-shadow: 0 -1px inset;\n",
       "}\n",
       "#h2o-table-2 .h2o-table tbody {\n",
       "  overflow: auto;\n",
       "}\n",
       "#h2o-table-2 .h2o-table th,\n",
       "#h2o-table-2 .h2o-table td {\n",
       "  text-align: right;\n",
       "  /* border: 1px solid; */\n",
       "}\n",
       "#h2o-table-2 .h2o-table tr:nth-child(even) {\n",
       "  /* background: #F5F5F5 */\n",
       "}\n",
       "\n",
       "</style>      \n",
       "<div id=\"h2o-table-2\" class=\"h2o-container\">\n",
       "  <table class=\"h2o-table\">\n",
       "    <caption>Model Summary: </caption>\n",
       "    <thead><tr><th></th>\n",
       "<th>number_of_trees</th>\n",
       "<th>number_of_internal_trees</th>\n",
       "<th>model_size_in_bytes</th>\n",
       "<th>min_depth</th>\n",
       "<th>max_depth</th>\n",
       "<th>mean_depth</th>\n",
       "<th>min_leaves</th>\n",
       "<th>max_leaves</th>\n",
       "<th>mean_leaves</th></tr></thead>\n",
       "    <tbody><tr><td></td>\n",
       "<td>20.0</td>\n",
       "<td>40.0</td>\n",
       "<td>152147.0</td>\n",
       "<td>15.0</td>\n",
       "<td>15.0</td>\n",
       "<td>15.0</td>\n",
       "<td>226.0</td>\n",
       "<td>375.0</td>\n",
       "<td>299.6</td></tr></tbody>\n",
       "  </table>\n",
       "</div>\n",
       "</div>\n",
       "<div style='margin: 1em 0 1em 0;'><pre style='margin: 1em 0 1em 0;'>ModelMetricsBinomialUplift: upliftdrf\n",
       "** Reported on train data. **\n",
       "\n",
       "AUUC: 27653.366951809396\n",
       "AUUC normalized: 0.835449152622639</pre>\n",
       "<div style='margin: 1em 0 1em 0;'>\n",
       "<style>\n",
       "\n",
       "#h2o-table-3.h2o-container {\n",
       "  overflow-x: auto;\n",
       "}\n",
       "#h2o-table-3 .h2o-table {\n",
       "  /* width: 100%; */\n",
       "  margin-top: 1em;\n",
       "  margin-bottom: 1em;\n",
       "}\n",
       "#h2o-table-3 .h2o-table caption {\n",
       "  white-space: nowrap;\n",
       "  caption-side: top;\n",
       "  text-align: left;\n",
       "  /* margin-left: 1em; */\n",
       "  margin: 0;\n",
       "  font-size: larger;\n",
       "}\n",
       "#h2o-table-3 .h2o-table thead {\n",
       "  white-space: nowrap; \n",
       "  position: sticky;\n",
       "  top: 0;\n",
       "  box-shadow: 0 -1px inset;\n",
       "}\n",
       "#h2o-table-3 .h2o-table tbody {\n",
       "  overflow: auto;\n",
       "}\n",
       "#h2o-table-3 .h2o-table th,\n",
       "#h2o-table-3 .h2o-table td {\n",
       "  text-align: right;\n",
       "  /* border: 1px solid; */\n",
       "}\n",
       "#h2o-table-3 .h2o-table tr:nth-child(even) {\n",
       "  /* background: #F5F5F5 */\n",
       "}\n",
       "\n",
       "</style>      \n",
       "<div id=\"h2o-table-3\" class=\"h2o-container\">\n",
       "  <table class=\"h2o-table\">\n",
       "    <caption>AUUC table (number of bins: 975): All types of AUUC value</caption>\n",
       "    <thead><tr><th>uplift_type</th>\n",
       "<th>qini</th>\n",
       "<th>lift</th>\n",
       "<th>gain</th></tr></thead>\n",
       "    <tbody><tr><td>AUUC value</td>\n",
       "<td>13950.3673552</td>\n",
       "<td>0.0255259</td>\n",
       "<td>27653.3669518</td></tr>\n",
       "<tr><td>AUUC normalized</td>\n",
       "<td>0.8429225</td>\n",
       "<td>0.0255259</td>\n",
       "<td>0.8354492</td></tr>\n",
       "<tr><td>AUUC random value</td>\n",
       "<td>8284.8809987</td>\n",
       "<td>0.0049387</td>\n",
       "<td>16569.7619974</td></tr></tbody>\n",
       "  </table>\n",
       "</div>\n",
       "</div>\n",
       "<pre style='margin: 1em 0 1em 0;'>Qini value: 5665.486356483165</pre>\n",
       "<div style='margin: 1em 0 1em 0;'>\n",
       "<style>\n",
       "\n",
       "#h2o-table-4.h2o-container {\n",
       "  overflow-x: auto;\n",
       "}\n",
       "#h2o-table-4 .h2o-table {\n",
       "  /* width: 100%; */\n",
       "  margin-top: 1em;\n",
       "  margin-bottom: 1em;\n",
       "}\n",
       "#h2o-table-4 .h2o-table caption {\n",
       "  white-space: nowrap;\n",
       "  caption-side: top;\n",
       "  text-align: left;\n",
       "  /* margin-left: 1em; */\n",
       "  margin: 0;\n",
       "  font-size: larger;\n",
       "}\n",
       "#h2o-table-4 .h2o-table thead {\n",
       "  white-space: nowrap; \n",
       "  position: sticky;\n",
       "  top: 0;\n",
       "  box-shadow: 0 -1px inset;\n",
       "}\n",
       "#h2o-table-4 .h2o-table tbody {\n",
       "  overflow: auto;\n",
       "}\n",
       "#h2o-table-4 .h2o-table th,\n",
       "#h2o-table-4 .h2o-table td {\n",
       "  text-align: right;\n",
       "  /* border: 1px solid; */\n",
       "}\n",
       "#h2o-table-4 .h2o-table tr:nth-child(even) {\n",
       "  /* background: #F5F5F5 */\n",
       "}\n",
       "\n",
       "</style>      \n",
       "<div id=\"h2o-table-4\" class=\"h2o-container\">\n",
       "  <table class=\"h2o-table\">\n",
       "    <caption>AECU values table: All types of AECU value</caption>\n",
       "    <thead><tr><th>uplift_type</th>\n",
       "<th>qini</th>\n",
       "<th>lift</th>\n",
       "<th>gain</th></tr></thead>\n",
       "    <tbody><tr><td>AECU value</td>\n",
       "<td>5665.4863565</td>\n",
       "<td>0.0205872</td>\n",
       "<td>11083.6049544</td></tr></tbody>\n",
       "  </table>\n",
       "</div>\n",
       "</div></div>\n",
       "<div style='margin: 1em 0 1em 0;'>\n",
       "<style>\n",
       "\n",
       "#h2o-table-5.h2o-container {\n",
       "  overflow-x: auto;\n",
       "}\n",
       "#h2o-table-5 .h2o-table {\n",
       "  /* width: 100%; */\n",
       "  margin-top: 1em;\n",
       "  margin-bottom: 1em;\n",
       "}\n",
       "#h2o-table-5 .h2o-table caption {\n",
       "  white-space: nowrap;\n",
       "  caption-side: top;\n",
       "  text-align: left;\n",
       "  /* margin-left: 1em; */\n",
       "  margin: 0;\n",
       "  font-size: larger;\n",
       "}\n",
       "#h2o-table-5 .h2o-table thead {\n",
       "  white-space: nowrap; \n",
       "  position: sticky;\n",
       "  top: 0;\n",
       "  box-shadow: 0 -1px inset;\n",
       "}\n",
       "#h2o-table-5 .h2o-table tbody {\n",
       "  overflow: auto;\n",
       "}\n",
       "#h2o-table-5 .h2o-table th,\n",
       "#h2o-table-5 .h2o-table td {\n",
       "  text-align: right;\n",
       "  /* border: 1px solid; */\n",
       "}\n",
       "#h2o-table-5 .h2o-table tr:nth-child(even) {\n",
       "  /* background: #F5F5F5 */\n",
       "}\n",
       "\n",
       "</style>      \n",
       "<div id=\"h2o-table-5\" class=\"h2o-container\">\n",
       "  <table class=\"h2o-table\">\n",
       "    <caption>Scoring History: </caption>\n",
       "    <thead><tr><th></th>\n",
       "<th>timestamp</th>\n",
       "<th>duration</th>\n",
       "<th>number_of_trees</th>\n",
       "<th>training_auuc_nbins</th>\n",
       "<th>training_auuc</th>\n",
       "<th>training_auuc_normalized</th>\n",
       "<th>training_qini_value</th></tr></thead>\n",
       "    <tbody><tr><td></td>\n",
       "<td>2023-07-20 15:32:20</td>\n",
       "<td> 0.087 sec</td>\n",
       "<td>0.0</td>\n",
       "<td>0</td>\n",
       "<td>nan</td>\n",
       "<td>nan</td>\n",
       "<td>nan</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:32:25</td>\n",
       "<td> 5.001 sec</td>\n",
       "<td>1.0</td>\n",
       "<td>16</td>\n",
       "<td>27654.7598311</td>\n",
       "<td>0.8354912</td>\n",
       "<td>145.1979468</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:32:41</td>\n",
       "<td>20.732 sec</td>\n",
       "<td>2.0</td>\n",
       "<td>36</td>\n",
       "<td>24922.9860290</td>\n",
       "<td>0.7529603</td>\n",
       "<td>659.4607747</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:32:56</td>\n",
       "<td>35.267 sec</td>\n",
       "<td>3.0</td>\n",
       "<td>61</td>\n",
       "<td>23986.4776199</td>\n",
       "<td>0.7246670</td>\n",
       "<td>1427.5581006</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:33:11</td>\n",
       "<td>51.081 sec</td>\n",
       "<td>4.0</td>\n",
       "<td>97</td>\n",
       "<td>23745.5319037</td>\n",
       "<td>0.7173877</td>\n",
       "<td>2139.2136934</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:33:28</td>\n",
       "<td> 1 min  7.611 sec</td>\n",
       "<td>5.0</td>\n",
       "<td>141</td>\n",
       "<td>23853.2737316</td>\n",
       "<td>0.7206427</td>\n",
       "<td>2742.8995838</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:33:46</td>\n",
       "<td> 1 min 25.335 sec</td>\n",
       "<td>6.0</td>\n",
       "<td>208</td>\n",
       "<td>24297.4650375</td>\n",
       "<td>0.7340624</td>\n",
       "<td>3312.8830197</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:34:04</td>\n",
       "<td> 1 min 43.525 sec</td>\n",
       "<td>7.0</td>\n",
       "<td>277</td>\n",
       "<td>24783.7052169</td>\n",
       "<td>0.7487524</td>\n",
       "<td>3782.8342577</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:34:26</td>\n",
       "<td> 2 min  6.050 sec</td>\n",
       "<td>8.0</td>\n",
       "<td>363</td>\n",
       "<td>25383.2162806</td>\n",
       "<td>0.7668645</td>\n",
       "<td>4234.3771787</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:34:50</td>\n",
       "<td> 2 min 29.392 sec</td>\n",
       "<td>9.0</td>\n",
       "<td>455</td>\n",
       "<td>25490.0203133</td>\n",
       "<td>0.7700912</td>\n",
       "<td>4383.0167770</td></tr>\n",
       "<tr><td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:35:39</td>\n",
       "<td> 3 min 18.196 sec</td>\n",
       "<td>11.0</td>\n",
       "<td>625</td>\n",
       "<td>26207.6358426</td>\n",
       "<td>0.7917715</td>\n",
       "<td>4852.3902988</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:36:04</td>\n",
       "<td> 3 min 43.319 sec</td>\n",
       "<td>12.0</td>\n",
       "<td>702</td>\n",
       "<td>26590.0001752</td>\n",
       "<td>0.8033233</td>\n",
       "<td>5071.4578797</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:36:30</td>\n",
       "<td> 4 min  9.613 sec</td>\n",
       "<td>13.0</td>\n",
       "<td>763</td>\n",
       "<td>26767.1653356</td>\n",
       "<td>0.8086757</td>\n",
       "<td>5177.9093412</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:36:57</td>\n",
       "<td> 4 min 36.351 sec</td>\n",
       "<td>14.0</td>\n",
       "<td>831</td>\n",
       "<td>27048.7667084</td>\n",
       "<td>0.8171833</td>\n",
       "<td>5333.6189983</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:37:26</td>\n",
       "<td> 5 min  6.015 sec</td>\n",
       "<td>15.0</td>\n",
       "<td>881</td>\n",
       "<td>27155.9905118</td>\n",
       "<td>0.8204227</td>\n",
       "<td>5397.0933351</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:37:59</td>\n",
       "<td> 5 min 39.043 sec</td>\n",
       "<td>16.0</td>\n",
       "<td>917</td>\n",
       "<td>27302.3346528</td>\n",
       "<td>0.8248439</td>\n",
       "<td>5475.8510860</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:38:30</td>\n",
       "<td> 6 min  9.563 sec</td>\n",
       "<td>17.0</td>\n",
       "<td>937</td>\n",
       "<td>27345.0732660</td>\n",
       "<td>0.8261351</td>\n",
       "<td>5501.8326709</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:39:04</td>\n",
       "<td> 6 min 43.276 sec</td>\n",
       "<td>18.0</td>\n",
       "<td>955</td>\n",
       "<td>27482.5051355</td>\n",
       "<td>0.8302872</td>\n",
       "<td>5575.6091664</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:39:38</td>\n",
       "<td> 7 min 17.728 sec</td>\n",
       "<td>19.0</td>\n",
       "<td>968</td>\n",
       "<td>27600.6538276</td>\n",
       "<td>0.8338566</td>\n",
       "<td>5636.6957599</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:40:18</td>\n",
       "<td> 7 min 57.455 sec</td>\n",
       "<td>20.0</td>\n",
       "<td>975</td>\n",
       "<td>27653.3669518</td>\n",
       "<td>0.8354492</td>\n",
       "<td>5665.4863565</td></tr></tbody>\n",
       "  </table>\n",
       "</div>\n",
       "<pre style='font-size: smaller; margin-bottom: 1em;'>[21 rows x 8 columns]</pre></div><pre style=\"font-size: smaller; margin: 1em 0 0 0;\">\n",
       "\n",
       "[tips]\n",
       "Use `model.explain()` to inspect the model.\n",
       "--\n",
       "Use `h2o.display.toggle_user_tips()` to switch on/off this section.</pre>"
      ],
      "text/plain": [
       "Model Details\n",
       "=============\n",
       "H2OUpliftRandomForestEstimator : Uplift Distributed Random Forest\n",
       "Model Key: UpliftDRF_model_python_1689859822235_1\n",
       "\n",
       "\n",
       "Model Summary: \n",
       "    number_of_trees    number_of_internal_trees    model_size_in_bytes    min_depth    max_depth    mean_depth    min_leaves    max_leaves    mean_leaves\n",
       "--  -----------------  --------------------------  ---------------------  -----------  -----------  ------------  ------------  ------------  -------------\n",
       "    20                 40                          152147                 15           15           15            226           375           299.6\n",
       "\n",
       "ModelMetricsBinomialUplift: upliftdrf\n",
       "** Reported on train data. **\n",
       "\n",
       "AUUC: 27653.366951809396\n",
       "AUUC normalized: 0.835449152622639\n",
       "\n",
       "AUUC table (number of bins: 975): All types of AUUC value\n",
       "uplift_type        qini      lift        gain\n",
       "-----------------  --------  ----------  --------\n",
       "AUUC value         13950.4   0.0255259   27653.4\n",
       "AUUC normalized    0.842922  0.0255259   0.835449\n",
       "AUUC random value  8284.88   0.00493868  16569.8\n",
       "\n",
       "Qini value: 5665.486356483165\n",
       "\n",
       "AECU values table: All types of AECU value\n",
       "uplift_type    qini     lift       gain\n",
       "-------------  -------  ---------  -------\n",
       "AECU value     5665.49  0.0205872  11083.6\n",
       "\n",
       "Scoring History: \n",
       "     timestamp            duration          number_of_trees    training_auuc_nbins    training_auuc       training_auuc_normalized    training_qini_value\n",
       "---  -------------------  ----------------  -----------------  ---------------------  ------------------  --------------------------  ---------------------\n",
       "     2023-07-20 15:32:20  0.087 sec         0.0                0                      nan                 nan                         nan\n",
       "     2023-07-20 15:32:25  5.001 sec         1.0                16                     27654.759831086874  0.8354912335675794          145.19794681488565\n",
       "     2023-07-20 15:32:41  20.732 sec        2.0                36                     24922.98602904925   0.7529603029924244          659.4607746892107\n",
       "     2023-07-20 15:32:56  35.267 sec        3.0                61                     23986.477619947054  0.7246669975814823          1427.558100645023\n",
       "     2023-07-20 15:33:11  51.081 sec        4.0                97                     23745.531903706986  0.7173876708068577          2139.2136934188948\n",
       "     2023-07-20 15:33:28  1 min  7.611 sec  5.0                141                    23853.273731581452  0.7206427109239109          2742.899583815666\n",
       "     2023-07-20 15:33:46  1 min 25.335 sec  6.0                208                    24297.46503748926   0.7340623878395549          3312.8830197257194\n",
       "     2023-07-20 15:34:04  1 min 43.525 sec  7.0                277                    24783.70521693464   0.7487524234723456          3782.8342577077\n",
       "     2023-07-20 15:34:26  2 min  6.050 sec  8.0                363                    25383.21628060475   0.7668645401995393          4234.377178706709\n",
       "     2023-07-20 15:34:50  2 min 29.392 sec  9.0                455                    25490.020313292083  0.7700912481357123          4383.016776975463\n",
       "---  ---                  ---               ---                ---                    ---                 ---                         ---\n",
       "     2023-07-20 15:35:39  3 min 18.196 sec  11.0               625                    26207.635842620588  0.7917714756078731          4852.390298847275\n",
       "     2023-07-20 15:36:04  3 min 43.319 sec  12.0               702                    26590.000175156056  0.8033232681316037          5071.457879669533\n",
       "     2023-07-20 15:36:30  4 min  9.613 sec  13.0               763                    26767.16533556117   0.8086756898961089          5177.909341153823\n",
       "     2023-07-20 15:36:57  4 min 36.351 sec  14.0               831                    27048.766708443647  0.8171832842430119          5333.618998341139\n",
       "     2023-07-20 15:37:26  5 min  6.015 sec  15.0               881                    27155.990511849228  0.8204226740739939          5397.093335071517\n",
       "     2023-07-20 15:37:59  5 min 39.043 sec  16.0               917                    27302.334652765978  0.8248439472134739          5475.851085983122\n",
       "     2023-07-20 15:38:30  6 min  9.563 sec  17.0               937                    27345.0732660454    0.8261351439892884          5501.832670900918\n",
       "     2023-07-20 15:39:04  6 min 43.276 sec  18.0               955                    27482.50513548014   0.8302871642139016          5575.609166375485\n",
       "     2023-07-20 15:39:38  7 min 17.728 sec  19.0               968                    27600.65382760647   0.8338566111059366          5636.695759858727\n",
       "     2023-07-20 15:40:18  7 min 57.455 sec  20.0               975                    27653.366951809396  0.835449152622639           5665.486356483165\n",
       "[21 rows x 8 columns]\n",
       "\n",
       "\n",
       "[tips]\n",
       "Use `model.explain()` to inspect the model.\n",
       "--\n",
       "Use `h2o.display.toggle_user_tips()` to switch on/off this section."
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ntree = 20\n",
    "max_depth = 15\n",
    "metric=\"Euclidean\"\n",
    "\n",
    "h2o_uplift_model = H2OUpliftRandomForestEstimator(\n",
    "                    ntrees=ntree,\n",
    "                    max_depth=max_depth,\n",
    "                    min_rows=30,\n",
    "                    nbins=1000,\n",
    "                    sample_rate=0.80,\n",
    "                    score_each_iteration=True,\n",
    "                    treatment_column=treatment_column,\n",
    "                    uplift_metric=metric,\n",
    "                    auuc_nbins=1000,\n",
    "                    auuc_type=\"gain\",\n",
    "                    seed=42)\n",
    "\n",
    "h2o_uplift_model.train(y=response_column, x=feature_cols, training_frame=h2o_train_df)\n",
    "h2o_uplift_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predict and plot Uplift Score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot uplift score\n",
    "# source https://www.kaggle.com/code/hughhuyton/criteo-uplift-modelling/notebook\n",
    "def plot_uplift_score(uplift_score):\n",
    "    plt.figure(figsize = (10,6))\n",
    "    plt.xlim(-.05, .1)\n",
    "    plt.hist(uplift_score, bins=1000, color=['#2077B4'])\n",
    "    plt.xlabel('Uplift score')\n",
    "    plt.ylabel('Number of observations in validation set')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "upliftdrf prediction progress: |█████████████████████████████████████████████████| (done) 100%\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class='dataframe'>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  uplift_predict</th><th style=\"text-align: right;\">  p_y1_with_treatment</th><th style=\"text-align: right;\">  p_y1_without_treatment</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">     0.000583659</td><td style=\"text-align: right;\">           0.00167115</td><td style=\"text-align: right;\">              0.00108749</td></tr>\n",
       "<tr><td style=\"text-align: right;\">     0.000583659</td><td style=\"text-align: right;\">           0.00167115</td><td style=\"text-align: right;\">              0.00108749</td></tr>\n",
       "<tr><td style=\"text-align: right;\">     0.000583659</td><td style=\"text-align: right;\">           0.00167115</td><td style=\"text-align: right;\">              0.00108749</td></tr>\n",
       "<tr><td style=\"text-align: right;\">     0.0441309  </td><td style=\"text-align: right;\">           0.0987045 </td><td style=\"text-align: right;\">              0.0545736 </td></tr>\n",
       "<tr><td style=\"text-align: right;\">     0.000583659</td><td style=\"text-align: right;\">           0.00167115</td><td style=\"text-align: right;\">              0.00108749</td></tr>\n",
       "<tr><td style=\"text-align: right;\">     0.000583659</td><td style=\"text-align: right;\">           0.00167115</td><td style=\"text-align: right;\">              0.00108749</td></tr>\n",
       "<tr><td style=\"text-align: right;\">     0.000583659</td><td style=\"text-align: right;\">           0.00167115</td><td style=\"text-align: right;\">              0.00108749</td></tr>\n",
       "<tr><td style=\"text-align: right;\">     0.00112479 </td><td style=\"text-align: right;\">           0.0346892 </td><td style=\"text-align: right;\">              0.0335644 </td></tr>\n",
       "<tr><td style=\"text-align: right;\">     0.000583659</td><td style=\"text-align: right;\">           0.00167115</td><td style=\"text-align: right;\">              0.00108749</td></tr>\n",
       "<tr><td style=\"text-align: right;\">     0.000583659</td><td style=\"text-align: right;\">           0.00167115</td><td style=\"text-align: right;\">              0.00108749</td></tr>\n",
       "</tbody>\n",
       "</table><pre style='font-size: smaller; margin-bottom: 1em;'>[2795919 rows x 3 columns]</pre>"
      ],
      "text/plain": [
       "  uplift_predict    p_y1_with_treatment    p_y1_without_treatment\n",
       "----------------  ---------------------  ------------------------\n",
       "     0.000583659             0.00167115                0.00108749\n",
       "     0.000583659             0.00167115                0.00108749\n",
       "     0.000583659             0.00167115                0.00108749\n",
       "     0.0441309               0.0987045                 0.0545736\n",
       "     0.000583659             0.00167115                0.00108749\n",
       "     0.000583659             0.00167115                0.00108749\n",
       "     0.000583659             0.00167115                0.00108749\n",
       "     0.00112479              0.0346892                 0.0335644\n",
       "     0.000583659             0.00167115                0.00108749\n",
       "     0.000583659             0.00167115                0.00108749\n",
       "[2795919 rows x 3 columns]\n"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "h2o_uplift_pred = h2o_uplift_model.predict(h2o_test_df)\n",
    "h2o_uplift_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAnIAAAF+CAYAAAAGBPdzAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAnR0lEQVR4nO3deZgldX3v8fdHQGRVltEowzADwbiLZgQTuWwqokQwiApuiHgxuRq36A3GBcQsLtHca1BxoiOQXHEhaMY4iriBUVEGBAQUGYHEIUSQQUBQZOB7/zjVyXHs7lM93dU91fN+Pc95uupX2/fUbJ+pql/9UlVIkiSpf+4z1wVIkiRpwxjkJEmSesogJ0mS1FMGOUmSpJ4yyEmSJPWUQU6SJKmn5l2QS7I8yY1JLm+5/nOTXJnkiiQf67o+SZKkmZL59h65JPsBPwfOqKpHjVh3T+CTwEFVdUuSB1bVjbNRpyRJ0nTNuytyVXU+sHa4LckeSb6Q5KIkX0/ysGbR/wTeX1W3NNsa4iRJUm/MuyA3gWXAn1TV7wKvBz7QtD8UeGiSbyS5IMkhc1ahJEnSFG0+1wV0Lcm2wO8Dn0oy1rxl83NzYE/gAGAhcH6SR1fVz2a5TEmSpCmb90GOwVXHn1XVXuMsWwN8u6ruBq5N8kMGwe7CWaxPkiRpg8z7W6tVdRuDkPYcgAw8tln8GQZX40iyM4NbrdfMQZmSJElTNu+CXJIzgW8Bv5NkTZLjgBcAxyW5FLgCOLxZ/Rzg5iRXAl8F3lBVN89F3ZIkSVM1714/IkmStKmYd1fkJEmSNhUGOUmSpJ6aV71Wd95551q8ePFclyFJkjTSRRdd9NOqWjCdfcyrILd48WJWrVo112VIkiSNlOTfprsPb61KkiT1lEFOkiSppwxykiRJPWWQkyRJ6imDnCRJUk8Z5CRJknrKICdJktRTBjlJkqSeMshJkiT1lEFOkiSppwxykiRJPWWQkyRJ6imDnCRJUk9tPtcFSJq6JSd87r+mr33HoXNYiSRpLnlFTpIkqacMcpIkST1lkJMkSeopg5wkSVJPGeQkSZJ6yiAnSZLUUwY5SZKknjLISZIk9ZRBTpIkqacMcpIkST1lkJMkSeqpzoJckl2TfDXJlUmuSPLqcdZJkvclWZ3ksiSPH1p2TJKrm88xXdUpSZLUV5t3uO91wJ9W1cVJtgMuSnJuVV05tM7TgT2bzz7AB4F9kuwInAgsBarZdkVV3dJhvZIkSb3S2RW5qrqhqi5upm8Hvg/sst5qhwNn1MAFwAOSPBh4GnBuVa1twtu5wCFd1SpJktRHs/KMXJLFwOOAb6+3aBfgx0Pza5q2idrH2/fxSVYlWXXTTTfNWM2SJEkbu86DXJJtgX8CXlNVt830/qtqWVUtraqlCxYsmOndS5IkbbQ6DXJJtmAQ4v5fVZ09zirXA7sOzS9s2iZqlyRJUqPLXqsBPgJ8v6reO8FqK4AXN71XnwjcWlU3AOcAByfZIckOwMFNmyRJkhpd9lp9EvAi4HtJLmna/hxYBFBVpwIrgWcAq4E7gWObZWuTvB24sNnu5Kpa22GtkiRJvdNZkKuqfwUyYp0CXjHBsuXA8g5KkyRJmhcc2UGSJKmnDHKSJEk9ZZCTJEnqKYOcJElSTxnkJEmSesogJ0mS1FMGOUmSpJ4yyEmSJPWUQU6SJKmnDHKSJEk9ZZCTJEnqKYOcJElSTxnkJEmSesogJ0mS1FMGOUmSpJ4yyEmSJPWUQU6SJKmnDHKSJEk9ZZCTJEnqKYOcJElSTxnkJEmSesogJ0mS1FMGOUmSpJ4yyEmSJPWUQU6SJKmnDHKSJEk9NTLIJdmyTZskSZJmV5srct9q2SZJkqRZtPlEC5L8FrALsFWSxwFpFm0PbD1qx0mWA38A3FhVjxpn+RuAFwzV8XBgQVWtTXIdcDtwD7Cuqpa2/kaSJEmbiAmDHPA04CXAQuC9Q+23AX/eYt+nAacAZ4y3sKreDbwbIMkzgddW1dqhVQ6sqp+2OI4kSdImacIgV1WnA6cneXZV/dNUd1xV5ydZ3HL1o4Ezp3oMSZKkTVmbZ+S+keQjST4PkOQRSY6bqQKSbA0cAgyHxQK+mOSiJMeP2P74JKuSrLrppptmqixJkqSNXpsg91HgHOAhzfwPgdfMYA3PBL6x3m3Vfavq8cDTgVck2W+ijatqWVUtraqlCxYsmMGyJEmSNm5tgtzOVfVJ4F6AqlrHoBPCTDmK9W6rVtX1zc8bgU8De8/g8SRJkuaFNkHujiQ7MbjdSZInArfOxMGT3B/YH/jnobZtkmw3Ng0cDFw+E8eTJEmaTybrtTrmdcAKYI8k3wAWAEeO2ijJmcABwM5J1gAnAlsAVNWpzWp/CHyxqu4Y2vRBwKeTjNX3sar6QqtvI0mStAkZGeSq6uIk+wO/w+BdcldV1d0ttju6xTqnMXhNyXDbNcBjR20rSZK0qWszRNdzgK2q6grgWcAnkjy+68IkSZI0uTbPyL2lqm5Psi/wZOAjwAe7LUuSJEmjtAlyYz1UDwX+vqo+B9y3u5IkSZLURpsgd32SDwHPA1Ym2bLldpIkSepQm0D2XAYvBH5aVf0M2BF4Q5dFSZIkabQ2vVbvBM4emr8BuKHLoiRJkjSat0glSZJ6yiAnSZLUUwY5SZKknmrzQuAjklyd5NYktyW5Pclts1GcJEmSJtZmrNV3Ac+squ93XYwkSZLaa3Nr9SeGOEmSpI1Pmytyq5J8AvgMcNdYY1WdPeEWkiRJ6lybILc9cCdw8FBbMfRuOUmSJM2+Ni8EPnY2CpEkSdLUtOm1ujDJp5Pc2Hz+KcnC2ShOkiRJE2vT2eGjwArgIc3ns02bJEmS5lCbILegqj5aVeuaz2nAgo7rkiRJ0ghtgtzNSV6YZLPm80Lg5q4LkyRJ0uTaBLmXAs8F/hO4ATgSsAOEJEnSHGvTa/XfgMNmoRZJkiRNwYRBLsn/rqp3Jfk7Bu+N+zVV9apOK5MkSdKkJrsiNzYs16rZKESSJElTM2GQq6rPNpN3VtWnhpcleU6nVUmSJGmkNp0d3tiyTZIkSbNosmfkng48A9glyfuGFm0PrOu6MEmSJE1usmfk/oPB83GHARcNtd8OvLbLoiRJkjTaZM/IXQpcmuRjVXX3LNYkSZKkFto8I7c4yVlJrkxyzdhn1EZJlie5McnlEyw/IMmtSS5pPm8dWnZIkquSrE5ywhS+jyRJ0iajTZD7KPBBBs/FHQicAfxji+1OAw4Zsc7Xq2qv5nMyQJLNgPcDTwceARyd5BEtjidJkrRJaRPktqqqLwOpqn+rqpOAQ0dtVFXnA2s3oKa9gdVVdU1V/Qr4OHD4BuxHkiRpXmsT5O5Kch/g6iSvTPKHwLYzdPzfS3Jpks8neWTTtgvw46F11jRt40pyfJJVSVbddNNNM1SWJEnSxq9NkHs1sDXwKuB3gRcBx8zAsS8GdquqxwJ/B3xmQ3ZSVcuqamlVLV2wYMEMlCVJktQPk71+BICqurCZ/Dlw7EwduKpuG5pemeQDSXYGrgd2HVp1YdMmSZKkIZO9EPizQE20vKoOm86Bk/wW8JOqqiR7M7g6eDPwM2DPJEsYBLijgOdP51iSJEnz0WRX5P6m+XkE8Fv8d0/Vo4GfjNpxkjOBA4Cdk6wBTgS2AKiqU4EjgT9Osg74BXBUVRWwLskrgXOAzYDlVXXFFL+XJEnSvDfZC4HPA0jynqpaOrTos0lWjdpxVR09YvkpwCkTLFsJrBx1DEmSpE1Zm84O2yTZfWymueW5TXclSZIkqY2RnR0YjKv6tWY0hwC7AS/vtCpJkiSN1KbX6heS7Ak8rGn6QVXd1W1ZkiRJGmWyXqsHVdVXkhyx3qI9klBVZ3dcmyRJkiYx2RW5/YGvAM8cZ1kBBjlJkqQ5NFmv1RObnzP2EmBJkiTNnMlurb5usg2r6r0zX44kSZLamuzW6nazVoUkSZKmbLJbq2+bzUIkSZI0NSNfP5LkfsBxwCOB+421V9VLO6xLkiRJI7QZ2eEfGIy1+jTgPGAhcHuXRUmSJGm0NkHut6vqLcAdVXU6cCiwT7dlSZIkaZQ2Qe7u5ufPkjwKuD/wwO5KkiRJUhttxlpdlmQH4C3ACmDbZlqSJElzqE2Q+2hV3cPg+bjdO65HkiRJLbW5tXptkmVJnpwknVckSZKkVtoEuYcBXwJeAVyX5JQk+3ZbliRJkkYZGeSq6s6q+mRVHQHsBWzP4DarJEmS5lCbK3Ik2T/JB4CLGLwU+LmdViVJkqSR2ozscB3wXeCTwBuq6o6ui5IkSdJobXqtPqaqbuu8EkmSJE1Jm2fkDHGSJEkboVbPyEmSJGnjY5CTJEnqqTadHbYEng0sHl6/qk7urixJkiSN0qazwz8DtzJ49chd3ZYjSZKkttoEuYVVdUjnlUiSJGlK2jwj980kj+68EkmSJE1JmyC3L3BRkquSXJbke0kuG7VRkuVJbkxy+QTLXzC0v28meezQsuua9kuSrGr/dSRJkjYdbW6tPn0D930acApwxgTLrwX2r6pbkjwdWAbsM7T8wKr66QYeW5Ikad6bMMgl2b55GfDtG7Ljqjo/yeJJln9zaPYCYOGGHEeSJGlTNdkVuY8Bf8Cgt2oBGVpWwO4zWMdxwOfX2/8XkxTwoapaNtGGSY4HjgdYtGjRDJYkSZK0cZswyFXVHzQ/l3RZQJIDGQS5fYea962q65M8EDg3yQ+q6vwJ6lzG4LYsS5curS5rlSRJ2pjM6cgOSR4DfBg4vKpuHmuvquubnzcCnwb2npsKJUmSNl5zFuSSLALOBl5UVT8cat8myXZj08DBwLg9XyVJkjZlbXqtbpAkZwIHADsnWQOcCGwBUFWnAm8FdgI+kARgXVUtBR4EfLpp2xz4WFV9oas6JUmS+qrNWKt7AGuq6q4kBwCPAc6oqp9Ntl1VHT1i+cuAl43Tfg3w2N/cQpIkScPa3Fr9J+CeJL/NoFPBrgx6tEqSJGkOtQly91bVOuAPgb+rqjcAD+62LEmSJI3SJsjdneRo4BjgX5q2LborSZIkSW20CXLHAr8H/GVVXZtkCfAP3ZYlSZKkUUZ2dqiqK4FXDc1fC7yzy6IkSZI0Wpteq08CTgJ2a9YPUFU1k0N0SZIkaYravEfuI8BrGYy5ek+35UiSJKmtNkHu1qr6/OjVJEmSNJvaBLmvJnk3g+G07hprrKqLO6tKkiRJI7UJcvs0P5cOtRVw0MyXI0mSpLba9Fo9cDYKkSRJ0tSMfI9ckvsneW+SVc3nPUnuPxvFSZIkaWJtXgi8HLgdeG7zuQ34aJdFSZIkabQ2z8jtUVXPHpp/W5JLOqpHkiRJLbW5IveLJPuOzTQvCP5FdyVJkiSpjTZX5P4YOL15Li7AWuAlXRYlSZKk0dr0Wr0EeGyS7Zv527ouSpIkSaNNGOSSvLCq/jHJ69ZrB6Cq3ttxbZIkSZrEZFfktml+bjfOsuqgFkmSJE3BhEGuqj7UTH6pqr4xvKzp8CBJkqQ51KbX6t+1bJMkSdIsmuwZud8Dfh9YsN5zctsDm3VdmCRJkiY32TNy9wW2bdYZfk7uNuDILouSJEnSaJM9I3cecF6S06rq32axJkmSJLXQ5oXAdyZ5N/BI4H5jjVV1UGdVSZIkaaQ2nR3+H/ADYAnwNuA64MIOa5IkSVILbYLcTlX1EeDuqjqvql4KeDVOkiRpjrW5tXp38/OGJIcC/wHs2F1JkiRJaqPNFbm/SHJ/4E+B1wMfBl7bZudJlie5McnlEyxPkvclWZ3ksiSPH1p2TJKrm88xbY4nSZK0KWlzRe7bVXUrcCtw4BT3fxpwCnDGBMufDuzZfPYBPgjsk2RH4ERgKYPhwC5KsqKqbpni8SVJkuatNlfkvpHki0mOS7LDVHZeVecDaydZ5XDgjBq4AHhAkgcDTwPOraq1TXg7FzhkKseWJEma70YGuap6KPBmBq8fuSjJvyR54Qwdfxfgx0Pza5q2idp/Q5Ljk6xKsuqmm26aobIkSZI2fm2uyFFV36mq1wF7M7jCdnqnVU1BVS2rqqVVtXTBggVzXY4kSdKsGRnkkmzfdDz4PPBN4AYGgW4mXA/sOjS/sGmbqF2SJEmNNlfkLgX2Ak6uqodW1Z9V1UUzdPwVwIub3qtPBG6tqhuAc4CDk+zQPJd3cNMmSZKkxqS9VpNsBpxdVX+6ITtPciZwALBzkjUMeqJuAVBVpwIrgWcAq4E7gWObZWuTvJ3/HkHi5KqarNOEJEnSJmfSIFdV9yT5/Q3deVUdPWJ5Aa+YYNlyYPmGHluSJGm+a/MeuUuSrAA+Bdwx1lhVZ3dWlSRJkkZqE+TuB9zMr4+vWoBBTpIkaQ6NDHJVdexsFCJJkqSpafP6kYcm+fLYeKlJHpPkzd2XJkmSpMm0ef3I3wNvBO4GqKrLgKO6LEqSJEmjtQlyW1fVd9ZrW9dFMZIkSWqvTZD7aZI9GHRwIMmRDEZ3kCRJ0hxq02v1FcAy4GFJrgeuBV7YaVWSJEkaqU2v1WuApyTZBrhPVd3efVmSJEkapU2v1Vcn2Z7BEFp/m+TiJAd3X5okSZIm0+YZuZdW1W0MBq7fCXgR8I5Oq5IkSdJIbYJcmp/PAM6oqiuG2iRJkjRH2gS5i5J8kUGQOyfJdsC93ZYlSZKkUdr0Wj0O2Au4pqruTLIT4LBdkiRJc6xNr9V7kywGXpikgH+tqk93XpkkSZIm1abX6geAPwK+B1wOvDzJ+7suTJIkSZNrc2v1IODhVTU2ssPpwJWdViVJkqSR2nR2WA0sGprfFbi6m3IkSZLU1oRX5JJ8lsH4qtsB30/ynWbR3sB3JtpOkiRJs2OyW6t/M2tVSJIkacomDHJVdd7YdJIHAU9oZr9TVTd2XZgkSZIm16bX6nMZ3Ep9DvBc4NtJjuy6MEmSJE2uTa/VNwFPGLsKl2QB8CXgrC4LkyRJ0uTa9Fq9z3q3Um9uuZ0kSZI61OaK3BeSnAOc2cw/D1jZXUmSJElqo80QXW9IcgSwb9O0zCG6JEmS5l6bK3JU1dnA2R3XIkmSpCnwWTdJkqSe6jTIJTkkyVVJVic5YZzlf5vkkubzwyQ/G1p2z9CyFV3WKUmS1EeTDdH15ap6cpJ3VtWfTXXHSTYD3g88FVgDXJhkRVVdObZOVb12aP0/AR43tItfVNVeUz2uJEnSpmKyZ+QenOT3gcOSfBzI8MKqunjEvvcGVlfVNQDNPg4Hrpxg/aOBE1tVLUmSpEmD3FuBtwALgfeut6yAg0bsexfgx0Pza4B9xlsxyW7AEuArQ833S7IKWAe8o6o+M8G2xwPHAyxatGhESZIkSfPHZGOtngWcleQtVfX2jus4Cjirqu4Zatutqq5PsjvwlSTfq6ofjVPnMmAZwNKlS6vjOiVJkjYabd4j9/YkhwH7NU1fq6p/abHv64Fdh+YXNm3jOQp4xXrHvb75eU2SrzF4fu43gpwkSdKmamSv1SR/DbyawbNtVwKvTvJXLfZ9IbBnkiVJ7ssgrP1G79MkDwN2AL411LZDki2b6Z2BJzHxs3WSJEmbpDYvBD4U2Kuq7gVIcjrwXeDPJ9uoqtYleSVwDrAZsLyqrkhyMrCqqsZC3VHAx6tq+Lbow4EPJbmXQdh8x3BvV0mSJLUc2QF4ALC2mb5/251X1UrWG5e1qt663vxJ42z3TeDRbY8jSZK0KWoT5P4a+G6SrzJ4Bcl+wG+83FeSJEmzq01nhzObzgZPaJr+rKr+s9OqJEmSNFKrW6tVdQPjdFSQJEnS3Ol0rFVJkiR1xyAnSZLUU5MGuSSbJfnBbBUjSZKk9iYNcs2QWVclcRBTSZKkjUybzg47AFck+Q5wx1hjVR3WWVWSJEkaqU2Qe0vnVUiSJGnK2rxH7rwkuwF7VtWXkmzNYMgtSZIkzaGRvVaT/E/gLOBDTdMuwGc6rEmSJEkttHn9yCuAJwG3AVTV1cADuyxKkiRJo7UJcndV1a/GZpJsDlR3JUmSJKmNNkHuvCR/DmyV5KnAp4DPdluWJEmSRmkT5E4AbgK+B7wcWAm8ucuiJEmSNFqbXqv3Jjkd+DaDW6pXVZW3ViVJkubYyCCX5FDgVOBHQIAlSV5eVZ/vujhJkiRNrM0Lgd8DHFhVqwGS7AF8DjDISZIkzaE2z8jdPhbiGtcAt3dUjyRJklqa8IpckiOayVVJVgKfZPCM3HOAC2ehNkmSJE1islurzxya/gmwfzN9E7BVZxVJkiSplQmDXFUdO5uFSJIkaWra9FpdAvwJsHh4/ao6rLuyJEmSNEqbXqufAT7CYDSHezutRpIkSa21CXK/rKr3dV6JJEmSpqRNkPu/SU4EvgjcNdZYVRd3VpUkSZJGahPkHg28CDiI/761Ws28JEmS5kibFwI/B9i9qvavqgObT6sQl+SQJFclWZ3khHGWvyTJTUkuaT4vG1p2TJKrm88x7b+SJEnSpqHNFbnLgQcAN05lx0k2A94PPBVYA1yYZEVVXbneqp+oqleut+2OwInAUgZX/y5qtr1lKjVIkiTNZ22C3AOAHyS5kF9/Rm7U60f2BlZX1TUAST4OHA6sH+TG8zTg3Kpa22x7LnAIcGaLbSVJkjYJbYLciRu4712AHw/NrwH2GWe9ZyfZD/gh8Nqq+vEE2+6ygXVIkiTNSyODXFWd1+HxPwucWVV3JXk5cDpT7ESR5HjgeIBFixbNfIWSJEkbqZGdHZLcnuS25vPLJPckua3Fvq8Hdh2aX9i0/Zequrmqxm7Xfhj43bbbDu1jWVUtraqlCxYsaFGWJEnS/DAyyFXVdlW1fVVtD2wFPBv4QIt9XwjsmWRJkvsCRwErhldI8uCh2cOA7zfT5wAHJ9khyQ7AwU2bJEmSGm1eP/JfauAzDDojjFp3HfBKBgHs+8Anq+qKJCcnGeso8aokVyS5FHgV8JJm27XA2xmEwQuBk8c6PkiSJGlg5DNySY4Ymr0Pg1eC/LLNzqtqJbByvba3Dk2/EXjjBNsuB5a3OY4kSdKmqE2v1WcOTa8DrmPwGhFJkiTNoTa9Vo+djUIkSZI0NRMGuSRvnWgZg8fl3t5BPZIkSWppsityd4zTtg1wHLATg84IkiRJmiMTBrmqes/YdJLtgFcDxwIfB94z0XaSJEmaHZM+I9cMXv864AUMRl14vAPXS5IkbRwme0bu3cARwDLg0VX181mrSpIkSSNN9kLgPwUeArwZ+I+hYbpubzlElyRJkjo02TNyUxr1QZIkSbPLsCZJktRTBjlJkqSeMshJkiT1lEFOkiSppwxykiRJPWWQkyRJ6imDnCRJUk8Z5CRJknrKICdJktRTBjlJkqSeMshJkiT1lEFOkiSppwxykiRJPWWQkyRJ6imDnCRJUk8Z5CRJknrKICdJktRTBjlJkqSeMshJkiT1VKdBLskhSa5KsjrJCeMsf12SK5NcluTLSXYbWnZPkkuaz4ou65QkSeqjzbvacZLNgPcDTwXWABcmWVFVVw6t9l1gaVXdmeSPgXcBz2uW/aKq9uqqPkmSpL7r8orc3sDqqrqmqn4FfBw4fHiFqvpqVd3ZzF4ALOywHkmSpHmlyyC3C/Djofk1TdtEjgM+PzR/vySrklyQ5Fkd1CdJktRrnd1anYokLwSWAvsPNe9WVdcn2R34SpLvVdWPxtn2eOB4gEWLFs1KvZIkSRuDLq/IXQ/sOjS/sGn7NUmeArwJOKyq7hprr6rrm5/XAF8DHjfeQapqWVUtraqlCxYsmLnqJUmSNnJdBrkLgT2TLElyX+Ao4Nd6nyZ5HPAhBiHuxqH2HZJs2UzvDDwJGO4kIUmStMnr7NZqVa1L8krgHGAzYHlVXZHkZGBVVa0A3g1sC3wqCcC/V9VhwMOBDyW5l0HYfMd6vV0l9ciSEz4HwLXvOHSOK5Gk+aXTZ+SqaiWwcr22tw5NP2WC7b4JPLrL2iRJkvrOkR0kSZJ6yiAnSZLUUwY5SZKknjLISZIk9ZRBTpIkqacMcpIkST1lkJMkSeopg5wkSVJPGeQkSZJ6yiAnSZLUUwY5SZKknjLISZIk9ZRBTpIkqacMcpIkST1lkJMkSeopg5wkSVJPGeQkSZJ6yiAnSZLUUwY5SZKknjLISZIk9ZRBTpIkqacMcpI0S5ac8DmWnPC5uS5D0jxikJMkSeopg5wkSVJPGeQkSZJ6yiAnSZLUUwY5SZKknjLISZIk9VSnQS7JIUmuSrI6yQnjLN8yySea5d9Osnho2Rub9quSPK3LOiVpOmb7tSK+xkTSmM272nGSzYD3A08F1gAXJllRVVcOrXYccEtV/XaSo4B3As9L8gjgKOCRwEOALyV5aFXd01W9kmbXWBC59h2Hzuj+ZnKf0jB/j2lj1FmQA/YGVlfVNQBJPg4cDgwHucOBk5rps4BTkqRp/3hV3QVcm2R1s79vdVivpB6aqytTs31cr8BJGk+XQW4X4MdD82uAfSZap6rWJbkV2Klpv2C9bXfprlRp4zPeFavx/jEf9Q/8xnTlYFStbb9z2+NM97tPJzxNtm0Xoazt74MNOfZk2070azXeuZ/s2NP9NZ/Or/VM/TpP9p03pj+Hml9SVd3sODkSOKSqXtbMvwjYp6peObTO5c06a5r5HzEIeycBF1TVPzbtHwE+X1VnjXOc44Hjm9nfAa7q5AttuJ2Bn851ET3geWrPc9WO56kdz1N7nqt2PE/t/U5VbTedHXR5Re56YNeh+YVN23jrrEmyOXB/4OaW2wJQVcuAZTNU84xLsqqqls51HRs7z1N7nqt2PE/teJ7a81y143lqL8mq6e6jy16rFwJ7JlmS5L4MOi+sWG+dFcAxzfSRwFdqcIlwBXBU06t1CbAn8J0Oa5UkSeqdzq7INc+8vRI4B9gMWF5VVyQ5GVhVVSuAjwD/0HRmWMsg7NGs90kGHSPWAa+wx6okSdKv6/LWKlW1Eli5Xttbh6Z/CTxngm3/EvjLLuubJRvtbd+NjOepPc9VO56ndjxP7Xmu2vE8tTftc9VZZwdJkiR1yyG6JEmSesogNwOS7Jjk3CRXNz93mGC9Y5p1rk5yzDjLVzSvZJmXpnOekmyd5HNJfpDkiiTvmN3qu+eQdu1t6LlK8tQkFyX5XvPzoFkvfhZN5/dUs3xRkp8nef2sFT0Hpvln7zFJvtX8vfS9JPeb1eJn2TT+7G2R5PTmHH0/yRtnvfhZ1OI87Zfk4iTrmte1DS+bNCv8hqryM80P8C7ghGb6BOCd46yzI3BN83OHZnqHoeVHAB8DLp/r77Mxnidga+DAZp37Al8Hnj7X32kGz81mwI+A3ZvvdynwiPXW+V/Aqc30UcAnmulHNOtvCSxp9rPZXH+njfRcPQ54SDP9KOD6uf4+G+N5Glp+FvAp4PVz/X02xvPE4Dnzy4DHNvM7+WdvwnP1fAYjNtH8fX4dsHiuv9McnqfFwGOAM4Ajh9onzQrjfbwiNzMOB05vpk8HnjXOOk8Dzq2qtVV1C3AucAhAkm2B1wF/0X2pc2qDz1NV3VlVXwWoql8BFzN4v+B88V9D2jXfb2xIu2HD5+8s4MnJrw9pV1XXAmND2s1XG3yuquq7VfUfTfsVwFZJtpyVqmffdH5PkeRZwLUMztN8Np3zdDBwWVVdClBVN9f8fsPCdM5VAdtk8M7YrYBfAbfNTtmzbuR5qqrrquoy4N71tp0wK0zEIDczHlRVNzTT/wk8aJx1xhuybGzYsbcD7wHu7KzCjcN0zxMASR4APBP4cgc1zpWR35v1hrQDhoe0G7XtfDKdczXs2cDFNRjTeT7a4PPU/Ofyz4C3zUKdc206v58eClSSc5rbZP97FuqdS9M5V2cBdwA3AP8O/E1Vre264Dkynb+Tp7xtp68fmU+SfAn4rXEWvWl4pqoqSeuuwEn2Avaoqteu/3xKH3V1nob2vzlwJvC+qrpmw6rUpi7JI4F3Mriiot90EvC3VfXz5gKdxrc5sC/wBAb/Ef9ykouqaj79J3Om7A3cAzyEwS3Dryf5kn+PT59BrqWqespEy5L8JMmDq+qGJA8GbhxnteuBA4bmFwJfA34PWJrkOga/Hg9M8rWqOoAe6vA8jVkGXF1V/2f61W5UZmVIu3liOueKJAuBTwMvrqofdV/unJnOedoHODLJu4AHAPcm+WVVndJ51bNvOudpDXB+Vf0UIMlK4PHMr7sFw6Zzrp4PfKGq7gZuTPINYCmDZ8Dmm+n8nTzq38Df4K3VmTE81NgxwD+Ps845wMFJdsigt+bBwDlV9cGqekhVLWbwP7sf9jXEtbDB5wkgyV8w+EvhNd2XOusc0q69DT5XzW35zzHodPON2Sp4jmzweaqq/1FVi5u/l/4P8FfzNMTB9P7snQM8OoNe9ZsD+zMYkWi+ms65+nfgIIAk2wBPBH4wK1XPvjbnaSIT/hs4obnu3TEfPgzu/38ZuBr4ErBj074U+PDQei9l8CD6auDYcfazmPnda3WDzxOD/5UU8H3gkubzsrn+TjN8fp4B/JBBb6c3NW0nA4c10/dj0INwNYOgtvvQtm9qtruKedSbd6bPFfBmBs/pXDL0eeBcf5+N7Tytt4+TmMe9Vqd7noAXMugQcjnwrrn+LhvruQK2bdqvYBB23zDX32WOz9MTGFzRvYPBFcsrhradNCus/3FkB0mSpJ7y1qokSVJPGeQkSZJ6yiAnSZLUUwY5SZKknjLISZIk9ZRBTlKvJFmc5PL12k5K8voR270kySnN9B8leXEz/bAklyT5bpI9kjy/u+olaWYZ5CRtcqrq1Ko6o5l9FnBWVT2OwdvYZyXINS+QlaRp8S8SSfNKkq8BlzJ4y/7mwEur6jvrrXMS8HMGLyZ9DXBPkicDWwEPT3IJcHpV/e3QNg8GPgFs3+z3j6vq60kOAf4K2Az4aVU9OcmOwHJgdwZjcB5fVZc1x92jaf/3JK8CTgUWNYd5Tc3/ESckzSCDnKT5aOuq2ivJfgwC1aPGW6mqViY5Ffh5Vf1NkgMYjGLwB+Os/nwGw+r9ZZLNgK2TLAD+Htivqq5tAhzA24DvVtWzkhwEnAHs1Sx7BLBvVf0iyccYDE7/r0kWMRiK5+Ez8P0lbSIMcpL6ZqLhaIbbzwSoqvOTbN+MsTpdFwLLk2wBfKaqLmmC3/lVdW1zvLXNuvsCz27avpJkpyTbN8tWVNUvmumnAI9IMnaM7ZNsW1U/n4F6JW0CDHKS+uZmYIf12nYErh2aXz/sTXsswiYU7gccCpyW5L3ALRuwqzuGpu8DPLGqfjnd+iRtmuzsIKlXmqtVNzS3LGluZx4C/OvQas9rlu0L3FpVt7bc/e3AduMtSLIb8JOq+nvgw8DjgQuA/ZIsGaoF4OvAC5q2Axg8O3fbOLv9IvAnQ8fYq2WdkgR4RU5SP70YeH9zVQzgbVX1o6Hlv0zyXWAL4KVT2O9lDDo+XAqcNtzZATgAeEOSuxl0lHhxVd2U5Hjg7CT3AW4EngqcxOA27GUMOjscM8HxXtV8j8sY/H18PvBHU6hX0iYuVdO+4yBJG42m1+rrq2rVXNciSV3z1qokSVJPeUVOkiSpp7wiJ0mS1FMGOUmSpJ4yyEmSJPWUQU6SJKmnDHKSJEk9ZZCTJEnqqf8PRHruv6+39xMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_uplift_score(h2o_uplift_pred['uplift_predict'].as_data_frame().uplift_predict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "perf_h2o = h2o_uplift_model.model_performance(h2o_test_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "### Area Under Uplift Curve (AUUC) calculation\n",
    "\n",
    "To calculate AUUC for big data, the predictions are binned to histograms. Due to this feature the results should be different compared to exact computation.\n",
    "\n",
    "To define AUUC, binned predictions are sorted from largest to smallest value. For every group the cumulative sum of observations statistic is calculated. The uplift is defined based on these statistics.\n",
    "\n",
    "\n",
    "#### Types of AUUC\n",
    "\n",
    "\n",
    "| AUUC type | &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp; &nbsp; &nbsp; Formula &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;|\n",
    "|:----------:|:-------------------------------------------:|\n",
    "| **Qini**   | $TY1 - CY1 * \\frac{T}{C}$                   |\n",
    "| **Lift**   | $\\frac{TY1}{T} - \\frac{CY1}{C}$             |\n",
    "| **Gain**   | $(\\frac{TY1}{T} - \\frac{CY1}{C}) * (T + C)$ |\n",
    "\n",
    "\n",
    "Where:\n",
    "\n",
    "- **T** how many observations are in the treatment group (how many data rows in the bin have treatment_column label == 1)\n",
    "- **C** how many observations are in the control group (how many data rows in the bin have treatment_column label == 0)\n",
    "- **TY1** how many observations are in the treatment group and respond to the offer (how many data rows in the bin have treatment_column label == 1 and response_column label == 1)\n",
    "- **CY1** how many observations are in the control group and respond to the offer (how many data rows in the bin have treatment_column label == 0 and response_column label == 1)\n",
    "\n",
    "\n",
    "The resulting AUUC value is:\n",
    "\n",
    "- Not normalized.\n",
    "- The result could be a positive or negative number.\n",
    "- Higher number means better model.\n",
    "\n",
    "More information about normalization is in **Normalized AUUC** section.\n",
    "\n",
    "\n",
    "For some observation groups the results should be NaN. In this case, the results from NaN groups are linearly interpolated to calculate AUUC and plot uplift curve.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "\n",
       "#h2o-table-6.h2o-container {\n",
       "  overflow-x: auto;\n",
       "}\n",
       "#h2o-table-6 .h2o-table {\n",
       "  /* width: 100%; */\n",
       "  margin-top: 1em;\n",
       "  margin-bottom: 1em;\n",
       "}\n",
       "#h2o-table-6 .h2o-table caption {\n",
       "  white-space: nowrap;\n",
       "  caption-side: top;\n",
       "  text-align: left;\n",
       "  /* margin-left: 1em; */\n",
       "  margin: 0;\n",
       "  font-size: larger;\n",
       "}\n",
       "#h2o-table-6 .h2o-table thead {\n",
       "  white-space: nowrap; \n",
       "  position: sticky;\n",
       "  top: 0;\n",
       "  box-shadow: 0 -1px inset;\n",
       "}\n",
       "#h2o-table-6 .h2o-table tbody {\n",
       "  overflow: auto;\n",
       "}\n",
       "#h2o-table-6 .h2o-table th,\n",
       "#h2o-table-6 .h2o-table td {\n",
       "  text-align: right;\n",
       "  /* border: 1px solid; */\n",
       "}\n",
       "#h2o-table-6 .h2o-table tr:nth-child(even) {\n",
       "  /* background: #F5F5F5 */\n",
       "}\n",
       "\n",
       "</style>      \n",
       "<div id=\"h2o-table-6\" class=\"h2o-container\">\n",
       "  <table class=\"h2o-table\">\n",
       "    <caption>AUUC table (number of bins: 113): All types of AUUC value</caption>\n",
       "    <thead><tr><th>uplift_type</th>\n",
       "<th>qini</th>\n",
       "<th>lift</th>\n",
       "<th>gain</th></tr></thead>\n",
       "    <tbody><tr><td>AUUC value</td>\n",
       "<td>21957.2812955</td>\n",
       "<td>0.0239431</td>\n",
       "<td>25778.1588999</td></tr>\n",
       "<tr><td>AUUC normalized</td>\n",
       "<td>0.8956995</td>\n",
       "<td>0.0239431</td>\n",
       "<td>0.8938296</td></tr>\n",
       "<tr><td>AUUC random value</td>\n",
       "<td>16257.7513719</td>\n",
       "<td>0.0068410</td>\n",
       "<td>19126.7594789</td></tr></tbody>\n",
       "  </table>\n",
       "</div>\n"
      ],
      "text/plain": [
       "AUUC table (number of bins: 113): All types of AUUC value\n",
       "uplift_type        qini      lift        gain\n",
       "-----------------  --------  ----------  -------\n",
       "AUUC value         21957.3   0.0239431   25778.2\n",
       "AUUC normalized    0.895699  0.0239431   0.89383\n",
       "AUUC random value  16257.8   0.00684096  19126.8"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "perf_h2o.auuc_table()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cumulative Uplift curve plot\n",
    "\n",
    "To plot the uplift curve, the ``plot_uplift``method can be used. There is specific parameter ``metric`` which can be ``\"qini\", \"gain\", or \"lift\"``. The most popular is the Qini uplift curve which is similar to the ROC curve. The Gain and Lift curves are also known from traditional binomial models. \n",
    "\n",
    "Depending on these curves, you can decide how many observations (for example customers) from the test dataset you send an offer to get optimal gain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAABHh0lEQVR4nO3deZzN1f/A8dfb2Jes5ZulLOlbyL61kKKSivSTpGxJyVKSFm2UkspSpKLSrlTiS6VNhhRCirQhyYQsYxv7mPfvj/MZbmOWO+Pe+7kz9/18PO7j3ns+2/vcy33P53zO5xxRVYwxxphok8/vAIwxxpj0WIIyxhgTlSxBGWOMiUqWoIwxxkQlS1DGGGOikiUoY4wxUckSlIlZIjJMRN7yO45giEgPEVkQ8D5JRKp5r4uIyCwR2SUi7/sXZXQQkdki0j3U65rIswRlIk5EuojIUu9HdpP3I3GB33FlRkReE5HHTmD7eBG5OU1ZSxFJyMn+VLW4qv7hve0IlAfKquq1wcQqzu0i8pOI7BWRBBF5X0TOyUk80URVL1fV10O9rok8S1AmokRkEPAMMAL3o3oa8DzQ3sewcrvTgd9VNTkb2zwL3AHcDpQBzgRmAFdk9+Aikj+72xgTFFW1hz0i8gBKAknAtZms8xrwWMD7lkBCwPs/gbuBFcBe4BVcopsN7AG+BEqnt23A9q2918OAtwKWvQ9sBnYB84FaXvktwGHgkBf/LK+8AjAN2AqsA27PpF7xwM1pytLWTXEJ4w9gG/A0kM9b1gNYkGbdM4BHvLgOe7Hdml6saY5bAzgCNAk23gyO3w9Y7dX9BWBUmn38DxiU3c8qiH9HccAo7zP6w4tDgfxpY0+N21t/h3fsyzP7XuwRPQ87gzKRdC5QGJh+gvv5P+AS3F/9V+GS0/3AybhWgdtzuN/ZuB/vU4DvgbcBVHWS9/opdU1rV4lIPmAW8CNQEWgFDBSRy3JaKU8HoBHQAHdWeVNmK6vqUNzZ6FQvtolpY01ns1a4xPjdCcZ6NdAUqAm8A1wnIgIgIqWBS4F3w/BZ9QauBOrjPquOWazfFPgNKAc8BbySGqeJbpagTCSVBbZp9pqi0jNeVf9R1b+Br4HFqrpcVQ/gkl/9nOxUVSer6h5VPYg7u6orIiUzWL0xcLKqPqqqh9RdD3oJ6JyTYwd4UlUTVfUvXFPo9Se4v/SUBTaFYD9PeLHux30PCjT3lnUEFqrqRkL/WXUCnlHVDaqaCDyRxfrrVfUlVT0CvA6cijvrNlHO2o5NJG0HyolI/hNMUv8EvN6fzvvi2d2hiMQBjwPX4s7EUrxF5XBNfmmdDlQQkZ0BZXG4H+r0JAMF0pQVwDXHBdoQ8Ho9rmks1LbjfqRP1NFYVVVF5F1cQp0PdAFSe0gG/VmJyGnAzwH7Te+7rMDxn1NmNgfsb5938pTtfyMm8uwMykTSQuAgrmkoI3uBogHv/3MCx/vXvrwkdHIG63bBNam1xl0rq5K6mfecdtj/DcA6VS0V8Cihqm0z2P9fAftMVZXjf1wrB7w+DdiYwf4yk9UUBXOASiLSKJN1gvke0h7nHaCjiJyOa1ab5pUH/Vmp6l9e02TxDJITuLO/tJ+TyYMsQZmIUdVdwMPABBG5WkSKikgBEblcRJ7yVvsBaCsiZUTkP8DAEzjk70BhEblCRAoADwKFMli3BC55bsf9MI9Is/wfoFrA+++APSJyr3cfUpyI1BaRxhnsfyrQU0SaeF28zwTuBN5Ns97dIlJaRCrjetlNDbKumcX6L6q6Gtdz8h2vq3tBESksIp1F5D5vtR+Aa7zv6AygV1YHVdXluI4LLwOfqepOb1F2P6usvAfcLiKVvGtd92W1gcmdLEGZiFLV0cAgXLLYivvruj+uizPAm7iL6X8Cn5OzH+jUY+0C+uJ+MP/GnRVkdN/RG7izmb9xTUyL0ix/BagpIjtFZIZ3PeNKoB6uZ1jqD3O616xU9TPcD+mruCbDT3DXQyalWfV/wDJcgvjYO252/SvWDNa5HXgOmADsBNbiOmjM8paPxfUE/MeL8+0gjz0FdxY6JbUgu59VEF4CPsP9O/ke+DCH+zFRTlRtwkJjooGIKFBDVdf4HUtuIiJVcImvQAg64JgoYmdQxhhjopIlKGOMMVHJmviMMcZEJTuDMsYYE5UsQRljjIlKlqBMnuZNc7FDRAqlKctw6gsRqSIimnaU7rTTWIjIqSLyijdlyB4R+VVEHhGRYtmI7xQReUdENnrzOX0jIk3THGOmt1y9HmuB21cUkf+JSKI3ZUafNMvVm04jyXu8HLBsdkB5kogcEpGVGcTZTES+8I6z1Zua49SA5YVE5EUR+cdbZ5aIVAxY3l/cFCsHReS1NPuu6S3b4T2+FJGawX6GJu+yBGXyLO/HvDluxIN2Id53GdzIGEWAc1W1BG4A21JA9WzsqjiwBGiIm/bideBjEUkdRSEF+BQ3QG563sJ1sS6PmypjhIhclGadugGjMxxNzOrmQgocteFb3Iju6SmNu2erCm7ooj24e7pS3YEbDLgObiiiHcD4gOUbgceAyenseyNu7L4yuKGlZnL8DcwmBtlYfCYv64a74XYx0J2Mf3xzYhDuR/pGVU0BUNUNuB/qoHkDp44JKJokIqOA/wLLVPUf4Pm0Z3MAXhJrCXRS1cPAjyLyAW4E9LnZiSMgmffIIM7ZadZ/DpgXUFQVN3rEP97yqYH1UtUPvfJGQKU0+96Ju1kYERHcVCBnZCd+kzfZGZTJy7rhRkB4G7hMREI5gnVr4MPU5JQeEVnhjeaQ3uP5DLapBxQEgrlZV9I8p76unWa9+SKyWUQ+TNtEGKAb8LWq/hnEcQFaAKsC3r8CnC8iFUSkKHADbvqSoIkbTPYA7swr7VBTJgZZgjJ5krgp5E8H3lPVZbihfLqE8BBZTlmhqnXSDJAa+OibTswn4YZ6esQbpilTqroH+AZ4yBtLrwGuKTBwkNcLcc1yZ+Ga0j5K72wMl6Bey+qYXpx1cGMq3h1QvBo3bNXfwG7gbODRYPaXSlVL4YY/6g8sz862Jm+yBGXyqu7A56q6zXs/xSuDrKe+SA4oy2idUE1ZAYCIFMGNg7dIVbOa3yjQDbjmtQ24WW3fImC8QVWd783BtBPX/FgVlzwCj30BbrTyD4KI8wzcmdEdqho4XcYE3EC8ZYFiuPHxsnUG5cW7F3gReENETsnu9iZvsQRl8hzvx74TcKHXtLUZN3J4XRGpS9ZTX2zCJaLM1vkS6CButtiM4liVppdc4OPFgPUK4QbLTcBN2R40VV2vqleq6smq2hTXySCzmXKVfzcJgkvcH6pqUmbHEjeNxpfAcFV9M83iesBr3gSGB3HNdE1EpFw2qpMqH+4ssGJWK5q8zRKUyYuuxl1or4n74ayHO2v4GteUlenUF97o29OAx0WkrLgpQa739pd6VjAGOAl43fvhTu3yPcZrAkNVawX2kkvz6ONtUwB35rIf6J7eNS0RKcyxaUIKee9Tl50tIiXETZlxI26a9THesloiUk/c9BbFgdG4JrhfArZPTeavZfaBel3GvwKeU9UX01llCdBNREp6deoLbEw9gxWR/F7ccUCc1ySZ31t2iYjU9+I8yYt/R2CcJkapqj3skaceuG7Zo9Mp74SbXTU/rqfbKtz1kjW4qTDyBaxbmmPTdOzAXes5P83+KuC6TW/G9ej7FRgKFM1GrBfizmr2AUkBj+YB62jaR8CygbhpS/YCC4BGAcsuBn7zlm3BnaXVSHP863FnhZJObKuAG7zXQ71jB8aYFLBuWVxnlC24HnkLgCYBy4elU49h3rJrvc8uyavLx0Adv/8d2cP/h43FZ4wxJipZE58xxpioZAnKGGNMVLIEZYwxJipZgjLGGBOVYm4svnLlymmVKlVyvP3evXspVizowapzLatn3hIr9YTYqWtequeyZcu2qerJactjLkFVqVKFpUuX5nj7+Ph4WrZsGbqAopTVM2+JlXpC7NQ1L9VTRNanV25NfMYYY6KSJShjjDFRKWwJSkQqi8hcEfnZG5PsDq98mIj8LSI/eI+2AdsMEZE1IvKbiFwWUN7GK1sjIvcFlFcVkcVe+VQRKRiu+hhjjImscJ5BJQN3qWpNoBnQL2Aa57GqWs97fAJu2megM1ALaIObpC1OROJwIyVfjhsL7fqA/Tzp7esM3HA0vcJYH2OMMREUtgSlqptU9Xvv9R7cwI+ZjU7cHnhXVQ+q6jrc+GhNvMcaVf1DVQ/hBvRs7828eTHHpgh4HTdIqDHGmDwgItegvFk86+Om3gbo7802OllESntlFXFz2qRK8MoyKi8L7FTV5DTlxhhj8oCwdzP3hvmfBgxU1d0i8gIwHDea8XDcFAA3hTmGW4BbAMqXL098fHyO95WUlHRC2+cWVs+8JVbqCbFTV7/rqeoe+cJ4mhPWBOXNCzMNeFtVPwRQ1X8Clr8EfOS9/RuoHLB5Ja+MDMq3A6VEJL93FhW4/r+o6iRgEkCjRo30RO4dyEv3HmTG6pm3xEo9IXbqGop6HjoEe/a4x+7d2Xv++msoVw4WLYLq1UNTp7TClqC8a0SvAL+o6piA8lNVdZP3tgPwk/d6JjBFRMbg5tmpgZsZVIAaIlIVl4A6A11UVUVkLtARd12qO/C/cNXHGGOiweHDLkls3lyYFSuyl1TSlh08GNwxixaFk06CEiWOPQNs27ad3r2L8uWXRcJyJhXOM6jzga7AShH5wSu7H9cLrx6uie9PvCmuVXWViLwH/IzrAdhP3cymiEh/4DPcbJyTVXWVt797gXdF5DFgOS4hGmNMVDl8OGdnKek9H0sqzTI9ZtGi/04oJ50ElSsfn2iyei5eHPKnyRRHjhwhLi6O8eNLMmTIF6xZczlnnhn6zy1sCUpVF+DOftL6JJNtHgceT6f8k/S2U9U/cL38jDEmpFKTSigSy4EDwR2zSJHjE0Tlyuknjo0bf6Vx47PSTSzpJZVQOHjwIM899xwvvvgiS5YsoX//UnTs2IZTTw39sSAGx+IzxuRdrvkrP+vXZz+JpC0LR1LJ7EylRInsJZX4+M20bHlWzj6obFJVZsyYwd13383atWtp06YNe/bsoVSpUpx6anrnIaFhCcoY46vk5NA1f7mkckGWxyxS5PgkUbEinH129pq/sptUcqO9e/dy5ZVXEh8fT82aNZk9ezZt2rSJyLHz+EdrjAmH1KQSisSyf39wxyxc+PgEUbEinHXWv8v/+WcNDRqckWliyetJJRT2799PkSJFKFasGKeffjrPP/88vXv3Jn8EPzz7moyJEdGUVII9Owl8XaBAcMeMj0+gZcszcv5Bxbj9+/czevRoxo4dy5IlS6hWrRqvvfaaL7FYgjImih05EpqEsnNn86C7FBcufHzCqFAh+72/spNUjP9UlXfeeYf77ruPDRs2cM011xAXF+drTJagjIkCqvDLLzBnjnusWgUbN8K+fcFtX6jQ8Qni1FPhzDPd+507N1KrVuWgEoslldiTnJzMRRddxIIFC6hfvz5vvvkmF154od9hWYIyxi8JCccS0pdfwibv9vXq1aFhQ2jXDkqWDK75q2AWE83Ex6+lZcvKma9kYs62bdsoV64c+fPnp3Xr1vTq1Ytu3bqRL5zjF2WDJShjImTnToiPd8lozhz49VdXfvLJ0KoVtG7tnqtU8TFIExP27NnDyJEjGTNmDJ9//jnNmzdn6NChfod1HEtQxoTJgQOwcKFLSF9+CUuXQkqKu8P/wguhd2+XlGrXDu+Am8akOnLkCK+99hoPPvggmzdv5oYbbqBKFP9FZAnKmBBJSYEffjiWkBYscL3d4uKgaVN48EGXkJo2zbpJzphQU1Vat25NfHw85557LjNmzKBp06Z+h5UpS1DG5JAqrF177BrSV19BYqJbVrs23HKLS0gtWrjrRcb4Yd26dZx++unky5ePbt260adPHzp16oQbzzu6WYIyJhv++cclotSktH69K69cGdq3d9eQLr6YsI1NZkywduzYwfDhw3nuued45ZVX6Nq1Kz179vQ7rGyxBGVMJpKSYP78Yx0bVqxw5aVKuUR0770uKdWoAbngD1ITAw4fPszEiRMZNmwYiYmJ9OrVi0suucTvsHLEEpQxAQ4fhu++O5aQFi50IzAUKgQXXABPPOESUoMG7tqSMdHmmmuu4aOPPuLiiy9mzJgx1K1b1++QcswSlIlpqvDTT8c6Nsyb586aRNy9SIMHu+tI553nBhg1JhqtWrWKqlWrUrRoUQYMGEDv3r256qqrcsV1psxYgjIx56+/jl1Dmj37PHbscOVnngldu7qE1LIllCnja5jGZGnr1q08/PDDTJo0ieHDh3P//fdz6aWX+h1WyFiCMnleYiLMnXssKa1e7crLl4eGDXfQpUt5WrWC007zN05jgnXw4EHeffdd3nnnHfbu3Uu/fv249dZb/Q4r5CxBmTxn/3745ptj15GWLXNNecWLuzOjvn3dWVKtWjBv3i+0bFne75CNyZZevXrx9ttvc8UVVzBq1CjOOisyExdGmiUok+sdOQLff3/sOtI338DBg27Q02bNYNgw17GhSRMbCNXkXsuWLaNChQqceuqp3HPPPdSrV4/Bgwf7HVZYWYIyuY6qa6ZLTUhz57px7gDq1IF+/dwZUvPm7qzJmNxs48aN3H///bzxxhv07duX5557jjp16pCYeld4HmYJyuQKmzcfu4b05ZduJHCA00+H//s/l5AuvhhOOcXfOI0JlX379jFq1CiefPJJkpOTufvuu7n//vv9DiuiLEGZqLR7t+vynZqUVq1y5WXKuOa61NG/q1WzG2RN3jRkyBDGjRtHx44defLJJ6lWrZrfIUWcJSgTFQ4dgkWLjnVsWLzYXVsqXNiNZdetm0tI9erZyN8m7/rmm28oWbIktWvX5p577qFjx440b97c77B8YwnK+CIlBVauPNZkN3++mz02Xz5o3Bjuu8+dJZ17rktSxuRl69at49577+X999+nU6dOTJ06lYoVK1KxYkW/Q/OVJSgTNqpuMNXU6Wb+/PNYQvrqK9i61ZWfdRbcdJM7Q7rwQjfOnTGxYPfu3YwYMYJnnnmGfPnyMXToUO6++26/w4oalqBMWKjCwIEwbhx06uTmSfr9d7esQgVo0+bYDLIx/keiiWETJkzgySefpGvXrowYMYJKlSr5HVJUsQRlQk7VjWE3bpx7/9577trRuHEuKZ11lnVsMLHryy+/RERo1aoVt99+O61bt6Zx48Z+hxWV7HKzCSlVuPNOGDMGbr/dXWvav9/dSDtgAJx9tiUnE5t+++03rrrqKi655BKefPJJAIoVK2bJKROWoEzIqMIdd8Czz7rmvWeeccmocGFLSiZ2JSYmMnDgQGrXrs28efMYOXIkM2fO9DusXCFsCUpEKovIXBH5WURWicgdXnkZEflCRFZ7z6W9chGRcSKyRkRWiEiDgH1199ZfLSLdA8obishKb5txktvHls/FVN0Z0vjxMGiQO4Oyb8MY+Pjjjxk/fjw33XQTq1ev5t5776WwdU0NSjjPoJKBu1S1JtAM6CciNYH7gDmqWgOY470HuByo4T1uAV4Al9CAoUBToAkwNDWpeev0DtiuTRjrYzKQkuKGF5owwV17GjXKkpOJXarKrFmzeOuttwC44YYb+Omnn5g4cSLly9vAxNkRtgSlqptU9Xvv9R7gF6Ai0B543VvtdeBq73V74A11FgGlRORU4DLgC1VNVNUdwBdAG2/ZSaq6SFUVeCNgXyZCUpPTCy/APffAU09ZcjKxa8WKFVxyySW0a9eO559/HlUlX758nH322X6HlitFpBefiFQB6gOLgfKquslbtBlI/ZOiIrAhYLMEryyz8oR0ytM7/i24szLKly9PfHx8juuSlJR0QtvnFsHUMyUFxow5k48/rkCXLutp02Yd8+ZFJr5Qse8z7/GjromJiUyePJnZs2dTvHhxBgwYQLt27ZgXxv8QMfGdqmpYH0BxYBlwjfd+Z5rlO7znj4ALAsrnAI2AwcCDAeUPeWWNgC8DypsDH2UVT8OGDfVEzJ0794S2zy2yqueRI6q9eqmC6v33q6akRCauULPvM+/xo67z5s3T/Pnz68CBA3X79u0ROWZe+k6BpZrO73VYz6BEpAAwDXhbVT/0iv8RkVNVdZPXTLfFK/8bqByweSWv7G+gZZryeK+8UjrrmzBLSYGbb4ZXX4WHHoJHHrFmPRNbVJX333+ftWvXMmTIEFq0aMH69eupUKGC36HlKeHsxSfAK8AvqjomYNFMILUnXnfgfwHl3bzefM2AXeqaAj8DLhWR0l7niEuBz7xlu0WkmXesbgH7MmFy5Aj06uWS09Ch8OijlpxMbPnuu++44IILuO6665g2bRqHDx8GsOQUBuHsxXc+0BW4WER+8B5tgZHAJSKyGmjtvQf4BPgDWAO8BPQFUNVEYDiwxHs86pXhrfOyt81aYHYY6xPzjhxxY+a99pqbpXbYMJ8DMiaCNm3aRNeuXWnatClr167lpZdeYvHixRSwaZrDJmxNfKq6AMjob+tW6ayvQL8M9jUZmJxO+VKg9gmEaYJ05Aj06AFvveXOmh56yO+IjImsffv2MXPmTIYMGcKQIUMoUaKE3yHleTYWn8lScjJ07w5TpsBjj8EDD/gdkTHhl5KSwptvvsk333zDpEmTqF69OgkJCZaYIsiGOjKZSk6Grl1dcnriCUtOJjbMnz+fxo0b06NHD3788Uf27NkDYMkpwixBmQwlJ8MNN8C778KTT7pJBI3JyzZt2kTHjh258MIL2bJlC2+99RYLFy60xOQTa+Iz6UpOFrp0gfffh6efdkMYGZPXFSlShKVLl/Loo49y1113UbRoUb9DimmWoMxxDh+G4cNrMn8+jB7tBn81Ji9KTk7m5ZdfZtq0aXz66aeUKlWK1atXW8+8KGFNfOZfDh2C666D+fNPZuxYS04m7/rss8+oV68et912G4cOHWL79u0AlpyiiCUoc9ShQ2569unToX//1Qwc6HdExoTetm3buOKKK2jTpg379+9n2rRpxMfHc8opp/gdmknDmvgMAAcPwrXXwqxZbk6n2rX/xs1gYkzekJKSQr58+ShZsiSJiYk8/fTTDBgwgEKFCvkdmsmAnUEZDh6Ejh1dcpowAfr39zsiY0Ln0KFDjBkzhlq1arFnzx4KFCjAt99+y+DBgy05RTlLUDHuwAG45hr46CM3p1Pfvn5HZExoqCozZsygVq1a3HXXXZx++uns2rULAJt8O3ewBBXDUpPTJ5/AxInQp4/fERkTGklJSbRq1YoOHTpQoEABZs+ezaeffkqlSpWy3thEDbsGFaP274cOHeCzz+Cll9z0GcbkdgcOHACgWLFi/Oc//+G5557j1ltvJX9++6nLjewMKgbt3w/t28Pnn8Mrr1hyMrnf/v37efzxx6lcuTKbN29GRJgyZQr9+vWz5JSLWYKKMarQuTN8+SVMnuymzzAmt1JV3nnnHc466ywefPBBmjdvbteX8hBLUDHmww9h5kw3fFGPHn5HY0zOHT58mBYtWtClSxfKli3L3Llz+fDDDylfvrzfoZkQyfDcV0SeUdWBIjIL0LTLVbVdWCMzIbdvnxsZok4duOMOv6MxJmcSExMpU6YMBQoU4MILL+Smm26iW7duxMXF+R2aCbHMGmff9J5HRSIQE35PPgl//QVvvgnWLG9ym6SkJEaOHMmYMWP46quvaNasGY899pjfYZkwyvBnSlWXec/zIheOCZetW2HUKDeUUYsWfkdjTPCOHDnC66+/zgMPPMDmzZvp0qULFStW9DssEwFZ/h0tIucDw4DTvfUFN0N7tfCGZkLl8GG45RbXe2/YML+jMSZ4qkqrVq2YN28ezZo1Y/r06TRr1szvsEyEBNPQ8wpwJ7AMOBLecEw4jBoFM2a4Jr6zz/Y7GmOytn79ek477TREhBtuuIE+ffpw3XXXWQ+9GBNML75dqjpbVbeo6vbUR9gjMyGRlOSmar/6arj7br+jMSZzO3fu5K677qJGjRpMnToVgN69e9O5c2dLTjEomDOouSLyNPAhcDC1UFW/D1tUJmTefBP27HEz4tr/bxOtkpOTmThxIkOHDiUxMZGbbrqJCy+80O+wjM+CSVBNvedGAWUKXBz6cEwo7d0LTz0FzZrBeef5HY0xGWvfvj2ffPIJF110EWPGjKFevXp+h2SiQJYJSlUvikQgJrRUXceI9evh1Vft7MlEn59//plq1apRuHBh+vXrxy233EK7du2sKc8cleE1KBG50XselN4jciGanHjsMZgyBR5+GFq29DsaY47ZunUrffv2pU6dOowbNw6Atm3b0r59e0tO5l8yO4Mq5j2XiEQgJnQOHIBnnoF27WDoUL+jMcY5ePAg48eP57HHHiMpKYnbbruNm2wwSJOJzG7Uneg9PxK5cEwozJgBiYkwYIA17Zno0aNHD959913atm3LqFGjONvueTBZCOZG3XGZLVfV20MXjgmFF16AqlXhYuvGYny2fPlyKlSoQPny5bn77rvp2bMnl156qd9hmVwimPugCgMNgNXeox5QEHfj7rKwRWZyZMYMmD8f+vWDfDZWvfHJxo0b6dmzJw0bNuSJJ54AoEGDBpacTLYE8xNWB2ipquNVdTzQCqinqq+r6usZbSQik0Vki4j8FFA2TET+FpEfvEfbgGVDRGSNiPwmIpcFlLfxytaIyH0B5VVFZLFXPlVECma38nmNKgwZAuecA/37+x2NiUX79u1j+PDhnHnmmUyZMoXBgwfzyCN2lcDkTDAJqjRwUsD74l5ZVl4D2qRTPlZV63mPTwBEpCbQGajlbfO8iMSJSBwwAbgcqAlc760L8KS3rzOAHUCvIGLK01asgF9/hdtug0KF/I7GxKJ77rmHhx9+mDZt2vDzzz/z1FNPUbJkSb/DMrlUMDfqjgSWi8hc3ECxLXCDx2ZKVeeLSJUg42gPvKuqB4F1IrIGaOItW6OqfwCIyLtAexH5BXejcBdvnde9mF4I8nh5kjcyDB07+huHiS3ffvstpUuX5uyzz+bee++lU6dOtLAh800IBHOj7qsiMptjI0rcq6qbT+CY/UWkG7AUuEtVdwAVgUUB6yR4ZQAb0pQ3BcoCO1U1OZ31jyMitwC3AJQvX574+PgcB5+UlHRC24fLypUnMWpUPWrW3MOqVctPeH/RWs9Qs3rm3ObNm5k4cSLx8fFcfPHFPPTQQ0eX+fmZ2neah6hq2B5AFeCngPflgThc0+LjwGSv/DngxoD1XgE6eo+XA8q7euuWw51ZpZZXDjxOZo+GDRvqiZg7d+4JbR8OCxeqliiheuaZqhs2hGaf0VjPcLB6Zt/u3bt1yJAhWqhQIS1SpIg+/PDDmpSUFLL9nyj7TnMfYKmm83sd0XlVVfWf1Nci8hLwkff2by/JpKrklZFB+XaglIjkV3cWFbh+TPniC+jQAf7zH/jqK7B53Ey4PfPMMzzxxBPceOONjBgxgsqVK2e9kTE5ENGOyCJyasDbDkBqD7+ZQGcRKSQiVYEawHfAEqCG12OvIK4jxUwv487FnWEBdAf+F4k6RJMPPoArroDq1WHBAktOJnzmzJnDvHlucu2BAweyePFi3nzzTUtOJqyCSlAicoGI9PRen+wlkay2eQdYCPxXRBJEpBfwlIisFJEVwEW4iRBR1VXAe8DPwKdAP1U94p0d9Qc+A34B3vPWBbgXGOR1qCiLaxaMGa+9BtddB02awLx57gzKmFD7/fffadeuHa1btz56P1OJEiVo0qRJFlsac+KCGUliKG6qjf8CrwIFgLeA8zPbTlWvT6c4wySiqo/jrkulLf8E+CSd8j841tMvpuzfDwMHwvnnw6efQtGifkdk8podO3bw6KOP8txzz1GkSBGeeOIJBg4c6HdYJsYEcw2qA1Af+B5AVTeKiA0g66P4eNi1Cx54wJKTCY/p06czbtw4evXqxfDhwylfvrzfIZkYFEyCOqSqKiIKICLFstrAhNfTT0ORIjYJoQkdVeXjjz8mKSmJzp070717d5o2bUqtWrX8Ds3EsGCuQb0nIhNxveZ6A18CL4U3LJORFStg7lzXxFfCzmNNCKxcuZJLL72Uq666ivHjx6OqxMXFWXIyvssyQanqKOADYBruOtTD6sbkMxF24ADceCOULw933ul3NCa327JlC7feeiv16tVj2bJlPPvss8THx9ukgSZqBNNJYhAwVVW/iEA8JhMPPggrV8JHH8HJJ/sdjcntVq5cyeTJk+nfvz9Dhw6lTJkyfodkzL8Ecw2qBPC5iCQCU4H3A2+4NZExdy6MGQN9+rh7n4zJLlXlgw8+4M8//+Tuu++mVatWrFu3jkqVKvkdmjHpCqaJ7xFVrQX0A04F5onIl2GPzBy1cyd07w5nnAGjRvkdjcmNfvvtN1q0aEGnTp147733SE52w1hacjLRLDsjSWwBNuOGGTolPOGY9PTvDxs3wltvQTHrQ2myYdOmTXTr1o0+ffrw+++/M2nSJBYtWkT+/BEd5cyYHAnmGlRfoBNwMvA+0FtVfw53YMaZOhXefhseecSNGmFMduzZs4fp06fTpUsXXnjhBU466aSsNzImSgTzZ1RlYKCq/hDmWEwaCQnumlPTpnD//X5HY3KDlJQU3nrrLRYvXsyECRM488wzSUhIYPny5ZacTK6TYROfiKT+a34a+EtEygQ+IhNe7EpJgR494NAhePNNsBYZk5UFCxbQtGlTunfvzpIlS9i7dy+AzWhrcq3MrkFN8Z6X4SYXXBbwWBrmuGLeuHEwZw6MHQs1avgdjYlmmzZt4tprr6V58+Zs3ryZN998k0WLFlHMLliaXC7Dv8tV9UrvOcuRy01o/fQT3HcfXHkl9O7tdzQm2hUqVIhFixbxyCOPMHjwYIraAI0mjwimk8QcVW2VVZkJjZQU6NoVTjoJXn4Z7KZ+k1ZycjKvvPIKM2bM4OOPP6ZMmTKsXbuWggUL+h2aMSGV2TWowt61pnIiUjrg+lMVwKbGC5NVq+CHH+Cxx9yQRsYE+uKLL6hfvz59+vRh7969bN++HcCSk8mTMrsGdSvuetNZ/Pv60/+A58IfWmx64w333KaNv3GY6LJ161auvPJKLr30Uvbt28cHH3zAvHnzONnGvDJ5WGbXoJ4FnhWRATY4bGS0awezZsHFF8Npp/kdjYkGKSkp5MuXj5IlS7Jlyxaeeuopbr/9dgoVKuR3aMaEXZbXoFR1vIjUBmoChQPK3whnYLFG1SUngA8/9DcW479Dhw7x/PPP88orrxztkbd48WIbadzElCyHOvKmfB/vPS4CngLahTmumLN5s3ueMAHstpXYparMnDmT2rVrc+edd1KhQgV27twJYMnJxJxgxuLrCLQCNqtqT6AuYD+hIbZqlXuuap36Y9bu3btp3bo17du3Jy4ujo8//phPP/2UihWtT5KJTcGMT7BfVVNEJNkbXWILbvgjE0Lffeeezz7b3zhM5B08eJBChQpRokQJypYty/jx47n11lspUKCA36EZ46tgEtRSESmFm+Z9GZAELAxnULHou+/g9NOhShW/IzGRsn//fsaOHcu4ceNYunQplSpV4r333vM7LGOiRjCdJPp6L18UkU+Bk1R1RXjDii07d7phja6/3u9ITCSoKlOnTuW+++5j/fr1tG/fnpSUFL/DMibqZJigRKRBZstU9fvwhBR7nn0WkpLgttv8jsSE26FDh2jVqhULFiygbt26vPrqq1x00UV+h2VMVMrsDGp0JssUuDjEscSknTvdgLBXXw316/sdjQmXnTt3UqpUKQoWLMj5559Pjx496NGjB3FxcX6HZkzUyuxGXfuzLgLGjoVdu2DYML8jMeGQlJTEk08+yZgxY5g3bx6NGjVi5MiRfodlTK4QzGCx3dIrtxt1T1xioktQ//d/ULeu39GYUEpJSeH111/ngQceYNOmTXTu3JnyNriiMdkSTC++xgGvC+PuifoesAR1gsaMcdee7Owpb1FVLrroIubPn0/Tpk2ZNm0a5557rt9hGZPrZHmjrqoOCHj0BhoAxbPaTkQmi8gWEfkpoKyMiHwhIqu959JeuYjIOBFZIyIrAjtoiEh3b/3VItI9oLyhiKz0thknuew2e1V46SV37al2bb+jMaGwYcMGVBURoXPnzrz99tssXLjQkpMxORTMSBJp7QWCGe/gNSDtmNz3AXNUtQYwx3sPcDlQw3vcArwALqEBQ4GmQBNgaGpS89bpHbBdrhr/e9Mm2LIFrANX7rdz504GDx5M9erV+dAbSPG2226jS5cuNjyRMScgmGtQs3C99sAltJpAlncTqup8b+6oQO2Blt7r14F44F6v/A1VVWCRiJQSkVO9db9Q1UQvli+ANiISj7sfa5FX/gZwNTA7q7iixULvVucGGXbmN9EuOTmZSZMmMXToULZv307Pnj0577zz/A7LmDwjmGtQowJeJwPrVTUhh8crr6qbvNebgdSrxhWBDQHrJXhlmZUnpFOeLhG5BXdmRvny5YmPj89h+K5X1olsn+qNN2pQpEh59u//hvh4zXqDCAtVPaPdidTz3nvv5bvvvqNevXqMGDGCGjVq8Ntvv/Hbb7+FNsgQiJXvE2KnrrFQz2BGkpgH4I3Dl997XSb1rCanVFVFJCK/zKo6CZgE0KhRI23ZsmWO9xUfH8+JbA+uaW/OHGjVClq3vvCE9hUuoahnbpDdev7yyy9Uq1aNQoUK8eCDD3LkyBHat28f9U15sfJ9QuzUNRbqGcx0G7eIyGZgBbAUNx7f0hwe7x+v6Q7veYtX/jf/HoC2kleWWXmldMpzhR9/hL17YdAgvyMxwdq2bRv9+vXjnHPOYcKECQBcddVVXH311VGfnIzJrYLpJHE3UFtVq6hqNVWtqqrVcni8mUBqT7zuuOnjU8u7eb35mgG7vKbAz4BLRaS01zniUuAzb9luEWnm9d7rFrCvqJfaAlS9ur9xmKwdOnSI0aNHc8YZZzBx4kT69OlD165d/Q7LmJgQzDWotcC+7O5YRN7BdXIoJyIJuN54I4H3RKQXsB7o5K3+CdAWWOMdqyeAqiaKyHBgibfeowFNi31xPQWL4DpH5JoOEm+/7abVqGyTlkS9rl278t5773H55ZczatQoatas6XdIxsSMYBLUEOBbEVkMHEwtVNXbM9tIVTMam7tVOusq0C+D/UwGJqdTvhTIdXcQbd4MixbB8OFgLUPRafny5VSuXJly5coxePBgevbsSZs2ueouBmPyhGCa+CYCXwGLcNefUh8mBz7+2D1fdZW/cZjjbdq0iZtuuomGDRvyxBNPANC4cWNLTsb4JJgzqAKqapfzQ0AVRo2CWrWgTh2/ozGp9u/fz+jRoxk5ciSHDh3irrvu4oEHHvA7LGNiXjAJarZ3H9Es/t3Ed0LdzGPR0qXw66/w8svWvBdNBg0axIsvvsg111zDU089RXXrvWJMVAgmQaVeSxoSUKZATnvyxawPPoACBdzo5cZfCxcupGzZsoC74bZz585ceGF03pNmTKwKZrDYquk8LDnlwBdfwLnnQqlSfkcSu9avX0/nzp0577zzGD58OABVqlSx5GRMFLL5oCLkxx9h+XJ4+mm/I4lNe/bsYeTIkYwePRoR4aGHHuKee+5h6dKc3nNujAk3mw8qQsaMgRIloFcvvyOJTaNGjWLEiBHccMMNPPHEE1S2m9CMiXrBjMU3IPC9iJQC3g1XQHlRSgrMng3t20Pp0lmvb0Jj7ty5FChQgAsuuIBBgwbRtm1bmjZt6ndYxpgghXM+KOP58UfYuhVat/Y7ktiwevVqrr76ai6++GJGjBgBQMmSJS05GZPLhG0+KHPMG29AXBy0bet3JHnbjh07GD58OM899xyFChVixIgR3HnnnX6HZYzJoUjPBxWTpk+Hdu3g5JP9jiRve//993nmmWfo1asXw4cP5z//+Y/fIRljTkCGCUpEzsBNMDgvTfn5IlJIVdeGPbo8YMMGWL8eBgzIel2TParK7Nmz2bt3L9deey033XQT5513HrVr57ohGo0x6cjsGtQzwO50ynd7y0wQpk1zz+3b+xtHXvPTTz9x2WWXccUVV/Dss8+iquTPn9+SkzF5SGYJqryqrkxb6JVVCVtEeciRI/DCC27cvTPO8DuavGHLli306dOHunXrsmTJEsaOHctXX31lkwYakwdldg2qVCbLioQ4jjzpgw/g99/dswmN5cuX8/LLL9OvXz+GDh16dLgiY0zek1mCWioivVX1pcBCEbkZm24jS6pu1Ihq1aBDB7+jyb1UlQ8//JC//vqLO++8k8suu4w//viD0047ze/QjDFhllmCGghMF5EbOJaQGgEFAfvJzcLatbBsGYwfD/lycreZYenSpQwaNIivv/6aRo0acfvttxMXF2fJyZgYkeFPp6r+o6rnAY8Af3qPR1T1XFXdHJnwcq/vvnPP55/vbxy50aZNm+jevTuNGzfm119/5cUXX2ThwoXExcX5HZoxJoKCGepoLjA3ArHkKZ99BmXL2sSEOZGYmMi0adO49957GTJkCCVLlvQ7JGOMD4K5UdfkwHffubMn+6M/aykpKbz99tssXbqUZ599llq1apGQkEApm5fEmJhmV0fC4O+/4bffoFEjvyOJfgsWLKBZs2Z069aNb7/9ln379gFYcjLGWIIKh1mzXC++a6/1O5LotXHjRjp16kTz5s3ZuHEjb7zxBosXL6Zo0aJ+h2aMiRLWxBcGCxe6cff++1+/I4leBQoUYMGCBQwbNozBgwdTrFgxv0MyxkQZS1AhtncvfPwxXHIJ2OAGxxw5coRXXnmFjz76iBkzZnDyySezbt06ChUq5HdoxpgoZU18Ifbkk7B9uw0OG2jOnDnUr1+fW2+9lR07drBjxw4AS07GmExZggqhgwfd2HtXXw3nned3NP7bsmULV111Fa1bt2bPnj28//77zJ8/34YnMsYExZr4QqhKFdi2Dfr29TsSf6kqIkLJkiVJSEhg5MiR3HHHHRQuXNjv0IwxuYglqBA5dAg2e+NrxOrU7ocPH+b555/n1Vdf5dtvv6Vo0aIsW7aMfDbWkzEmB+yXI0S++so9z5oVe50jVJVZs2ZRu3ZtBg4cyCmnnMLOnTsBLDkZY3LMl18PEflTRFaKyA8istQrKyMiX4jIau+5tFcuIjJORNaIyAoRaRCwn+7e+qtFpLsfdUk1fToULx57Z0+7du3ikksuoV27dogIH330EZ999hkVKlTwOzRjTC7n55+3F6lqPVVNHW/hPmCOqtYA5njvAS4HaniPW4AXwCU0YCjQFGgCDE1NapF25AjMmAFt20KsXGY5dOgQACeddBIlSpRg3LhxrFy5kiuuuMImDzTGhEQ0tb+0B173Xr8OXB1Q/oY6i4BSInIqcBnwhaomquoO4AugTYRjBtyNuVu2wDXX+HH0yDpw4AAjR46kSpUqbNq0CRFh+vTpDBgwgAIFCvgdnjEmD/Grk4QCn4uIAhNVdRJuivlN3vLNQHnvdUVgQ8C2CV5ZRuXHEZFbcGdflC9fnvj4+BwHnpSUdNz2EyZUp0CBihQv/g3x8UdyvO9okraeqkp8fDyTJk1i8+bNnHfeecyfP5/y5ctnvJNcIL3vMy+KlXpC7NQ1JuqpqhF/ABW951OAH4EWwM406+zwnj8CLggon4ObOHEw8GBA+UPA4KyO3bBhQz0Rc+fO/df7lBTVqlVV27Y9od1GncB6Hjx4UJs3b66A1qlTR+fMmeNfYCGW9vvMq2KlnqqxU9e8VE9gqabze+1LE5+q/u09bwGm464h/eM13eE9b/FW/xuoHLB5Ja8so/KI+uUXWLcO2reP9JHDb9euXQAULFiQxo0b89JLL/H9999z8cUX+xyZMSYWRDxBiUgxESmR+hq4FPgJmAmk9sTrDvzPez0T6Ob15msG7FLXFPgZcKmIlPY6R1zqlUXUypXuuVmzSB85fPbu3curr75KpUqV+OGHHwAYPXo0N998s81qa4yJGD+uQZUHpns9vfIDU1T1UxFZArwnIr2A9UAnb/1PgLbAGmAf0BNAVRNFZDiwxFvvUVVNjFw1nD/+cM/VqkX6yKGXkpLCm2++yf3338/GjRu57rrrKFOmjN9hGWNiVMQTlKr+AdRNp3w70CqdcgX6ZbCvycDkUMeYHb//Dv/5j7sHKjdLSUmhZcuWfP311zRp0oQhQ4bQv39/v8MyxsSwaOpmniutWgW1avkdRc79/fff7mJkvnxce+21vPXWWyxcuJDatWv7HZoxJsZZgjoBqvDrr3D22X5Hkn27du3i7rvvplq1asyaNQuAAQMGcMMNN9jwRMaYqGCDxZ6AzZthz57cNXNucnIyL7/8Mg899BDbt2+ne/fuNGrUKOsNjTEmwixBnYA1a9xzjRr+xpEdV1xxBZ9//jktWrRg7NixNGjQIOuNjDGZOnz4MAkJCRw4cCBixyxZsiS//PJLxI4XCoULF6ZSpUpBjzpjCeoErF/vnqtU8TWMLP32229UrVqVggUL0qdPH2699VY6dOhgY+YZEyIJCQmUKFGCKlWqROz/1Z49eyhRokREjhUKqsr27dtJSEigatWqQW1jFxtOwLp17vm00/yNIyPbtm1jwIAB1KpVi4kTJwLQoUMHrrnmGktOxoTQgQMHKFu2rP2/yoSIULZs2WydZdoZ1An46iuoXRuKFPE7kn87dOgQEyZM4NFHH2XPnj3ccsstdO7c2e+wjMnTLDllLbufkSWoHFKF77+Hrl39juR4Xbp0Ydq0aVx22WWMHj2aWrm5H7wxJmZZE18O7dgBu3dHzwgSP/zwA4mJbiCNwYMH88knn/Dpp59acjLGpOu11147ejP+iy++yBtvvAHAr7/+Sr169ahfvz5r165lypQpvsVoCSqH1q51z0Fe6wubTZs20atXLxo0aMBTTz0FQLNmzbj88sv9DcwYk2v06dOHbt26ATBjxgw6duzI8uXL2bBhg68Jypr4cih1kFi/BlzYv38/Y8eOZcSIERw6dIhBgwZx3333Zb2hMSasBg4Eb4zlkKlXD555JvN1/vzzT6688kp++uknAEaNGnV0zqi6desyb948kpOTmTx5Mk2aNPnXtsOGDaN48eLUrFmTZ555hri4OObMmcP+/fv55ZdfqFevHt27d+fOO+8MbcWyYAkqh1asgKJFoXp1f45/xx138NJLL9GhQweeeuopzjjjDH8CMcZEvX379vHDDz8wf/58brrppqNJLK22bdvSp08fihcvzuDBg4mPj2fUqFF89NFHEY7YsQSVQz/+COecA5EcFWjRokWcfPLJVK9enXvvvZcuXbrQsmXLyAVgjMlSVmc6frj++usBaNGiBbt372bnzp3+BhQkuwaVA6ruDKpOncgc76+//qJLly6ce+65DB8+HIDq1atbcjLGHJU/f35SUlKOvg+83yht9+7c0iXeElQObNgAiYlQ97hJQ0IrKSmJBx98kP/+979Mnz6dBx54gPHjx4f3oMaYXKl8+fJs2bKF7du3c/DgwX81y02dOhWABQsWULJkSUqWLBnUPkuUKMGePXvCEm8wrIkvB1591T23Om72qtAaOXIkjz/+OF26dOGJJ57gtGgdssIY47sCBQrw8MMP06RJEypWrMhZZ511dFnhwoWpX78+hw8fZvLk4KfQq1OnDnFxcdStW5cePXpEvJMEqhpTj4YNG+qJmDt3rp55puqll57QbjLd/8KFC1VVNTEx8ejrSJs7d64vx400q2fe40ddf/7554gfc/fu3UGtd+GFF+qSJUvCHE3w0vusgKWazu+1NfFl08GD+Vi9Gs49N7T7XbNmDR06dOCiiy7i8ccfB6B06dI0a9YstAcyxphcwpr4smnjxsKohm4OqJ07dzJ8+HDGjx9PwYIFefzxxyN/Gm2MybPi4+P9DiHHLEFl0/bthQCoVCk0+5syZQpjx46lZ8+ePPbYY5x66qmh2bExxuRylqCyadu2ggBUqJDzfcyePZuDBw9y9dVX07t3by644ALqRKrPujHG5BJ2DSqbtm1zZ1A5OdFZtWoVl19+OW3btmXMmDGoKgUKFLDkZIwx6bAElU0bNhSlQgU3zFGwtm7dSt++falbty6LFi1i9OjRfPnll7nmZjljjPGDJahs+uuvopx9dva2Wbx4MZMmTeK2225j9erVDBo0iIIFC4YnQGOMCYEqVaqwbds2X2Owa1DZcOgQrF1bnCuvzHw9VWX69Ols3LiR/v37c8UVV7B69Wqq+j03hzEmJhy9jyiSg4WGgSWobPjtNzh8OB8NGmS8zvfff8+gQYOYN28ejRo14rbbbiMuLs6SkzExJL1xMjt16kTfvn3Zt28fbdu2PW55jx496NGjB9u2baNjx47/WhZMV/E///yTyy67jKZNm7Js2TKaNGnCypUr2b9/Px07duSRRx4B3JlR9+7dmTVrFocPH+b999/nrLPOYvv27Vx//fX8/fffnHvuubj7Z50xY8YcHYHi5ptvZuDAgfz555+0adOGZs2a8e2339K4cWN69uzJ0KFD2bJlC2+//fZx03pkV+5OrxH288/uOb1Jajdt2kTPnj1p1KgRq1at4oUXXmDhwoXExcVFNkhjTMxavXo1ffv2ZdWqVYwePZqlS5eyYsUK5s2bx4oVK46uV65cOb7//ntuu+02Ro0aBcAjjzzCBRdcwKpVq+jQoQN//fUXAMuWLePVV19l8eLFLFq0iJdeeonly5cDboCBu+66i19//ZVff/2VKVOmsGDBAkaNGsWIESNOuD52BpUNv/4KIkqNGsd3bti6dStTp05l8ODBPPDAA0EPxmiMyXsyO+MpWrRopsvLlSuX45trTz/99KOjz7z33ntMmjSJ5ORkNm3axM8//3y0x/A111wDQMOGDfnwww8BmD9//tHXV1xxBaVLlwbcALMdOnSgWLFiR7f9+uuvadeuHVWrVuWcc84BoFatWrRq1QoR4ZxzzuHPP//MUR0C5foEJSJtgGeBOOBlVR0ZrmOtXQsnn3yQIkUKk5KSwjvvvMPy5csZNWoUderUISEhgTJlyoTr8MYYk6nUJLJu3TpGjRrFkiVLKF26ND169PjX9BuFCrnbZeLi4khOTs7x8VL3A5AvX76j7/Ply3dC+z26zxPeg49EJA6YAFwO1ASuF5Ga4TremjVQocJ+vv32W84991xuvPFG4uPj2b9/P4AlJ2NMVNi9ezfFihWjZMmS/PPPP8yePTvLbVq0aMGUKVMAN5jAjh07AGjevDkzZsxg37597N27l+nTp9O8efOwxp8qVycooAmwRlX/UNVDwLtA+3AdrHfvjRw50oXzzz+fhIQEXnvtNb777juKFCkSrkMaY0y21a1bl/r163PWWWfRpYv7zcrK0KFDmT9/PrVq1eLDDz88Or1PgwYN6NGjB02aNKFp06bcfPPN1K9fP9xVcNIb4jy3PICOuGa91Pddgecy2+ZEptvYtGmTli5dWh9++GFNSkrK8X5yg1iZnsHqmffYdBvRLTvTbYgGdCXMbUSkI9BGVW/23ncFmqpq/zTr3QLcAlC+fPmG7777bo6PmZiYGBNNeUlJSRQvXtzvMMLO6pn3+FHXkiVLcsYZZ0T0mEeOHMmVvYTXrFnDrl27/lV20UUXLVPVRmnXze2dJP4GKge8r+SV/YuqTgImATRq1EjTu0chWPHx8ene45DXWD3zllipJ/hT119++YUSJUpE9Jh79uyJ+DFDIXV232Dk9mtQS4AaIlJVRAoCnYGZPsdkjIlBubk1KlKy+xnl6gSlqslAf+Az4BfgPVVd5W9UxphYU7hwYbZv325JKhOqyvbt2ylcuHDQ2+T2Jj5U9RPgE7/jMMbErkqVKpGQkMDWrVsjdswDBw5k68c+GhQuXJhK2ZjtNdcnKGOM8VuBAgUiPt5mfHx85Lp7+yRXN/EZY4zJuyxBGWOMiUqWoIwxxkSlXH2jbk6IyFZg/Qnsohzg7zSTkWH1zFtipZ4QO3XNS/U8XVVPTlsYcwnqRInI0vTueM5rrJ55S6zUE2KnrrFQT2viM8YYE5UsQRljjIlKlqCyb5LfAUSI1TNviZV6QuzUNc/X065BGWOMiUp2BmWMMSYqWYIyxhgTlSxBpUNE2ojIbyKyRkTuS2d5IRGZ6i1fLCJVfAgzJIKoaw8R2SoiP3iPm/2I80SIyGQR2SIiP2WwXERknPcZrBCRBpGOMRSCqGdLEdkV8F0+HOkYQ0FEKovIXBH5WURWicgd6ayTV77TYOqaJ77XdKU3zW4sP4A4YC1QDSgI/AjUTLNOX+BF73VnYKrfcYexrj2A5/yO9QTr2QJoAPyUwfK2wGxAgGbAYr9jDlM9WwIf+R1nCOp5KtDAe10C+D2df7d55TsNpq554ntN72FnUMdrAqxR1T9U9RDwLtA+zTrtgde91x8ArUREIhhjqART11xPVecDiZms0h54Q51FQCkROTUy0YVOEPXME1R1k6p+773eg5sLrmKa1fLKdxpMXfMsS1DHqwhsCHifwPH/II6uo27SxF1A2YhEF1rB1BXg/7xmkg9EpHJkQouoYD+HvOBcEflRRGaLSC2/gzlRXvN6fWBxmkV57jvNpK6Qx77XVJagTFZmAVVUtQ7wBcfOHE3u8z1uzLO6wHhghr/hnBgRKQ5MAwaq6m6/4wmnLOqap77XQJagjvc3EHiWUMkrS3cdEckPlAS2RyS60Mqyrqq6XVUPem9fBhpGKLZICuY7z/VUdbeqJnmvPwEKiEg5n8PKEREpgPvBfltVP0xnlTzznWZV17z0vaZlCep4S4AaIlJVRAriOkHMTLPOTKC797oj8JV6VytzmSzrmqbdvh2uDTyvmQl083p+NQN2qeomv4MKNRH5T+q1UhFpgvv/n+v+sPLq8Arwi6qOyWC1PPGdBlPXvPK9psemfE9DVZNFpD/wGa6X22RVXSUijwJLVXUm7h/MmyKyBndRurN/EedckHW9XUTaAcm4uvbwLeAcEpF3cD2dyolIAjAUKACgqi8Cn+B6fa0B9gE9/Yn0xARRz47AbSKSDOwHOufSP6zOB7oCK0XkB6/sfuA0yFvfKcHVNa98r8exoY6MMcZEJWviM8YYE5UsQRljjIlKlqCMMcZEJUtQxhhjopIlKGOMMTmS1QDF6azfKWDg2ylZrW8JypgAIqIiMjrg/WARGRaifb8mIh1DtK/F3sjVf8m/R5uvEor9p3O8+3OwTQ8ReS4c8Zio8RrQJpgVRaQGMAQ4X1VrAQOz2sYSlDH/dhC4JtruxPdGLDlKVZuqaj3gYdxo+vW8x5/Z2U82ZDtBmbwvvQGKRaS6iHwqIstE5GsROctb1BuYoKo7vG23ZLV/S1DG/FsyMAm4M+2CtGdAIpLkPbcUkXki8j8R+UNERorIDSLynYisFJHqAbtpLSJLReR3EbnS2z5ORJ4WkSXeoLy3Buz3axGZCfycVeAicpV3ZrVcRL4UkfJe+TAReVNEvsHdYH6yiHzhNbO8LCLrUxOyiNzoxf2DiEz0YhsJFPHK3s5oPa+8p1e373A3mZrYMwkYoKoNgcHA8175mcCZIvKNiCwSkSzPvCxBGXO8CcANIlIyG9vUBfoAZ+Pu/D9TVZvgxi8cELBeFdw0J1cAL4pIYaAXbiiexkBjoLeIVPXWbwDcoapnBhHDAqCZqtbHTZ1yT8CymkBrVb0eN8LEV14zywd4oxKIyNnAdbgmmHrAEeAGVb0P2O+dod2Q0XrihsV6BJeYLvCOaWKIuEFtzwPe90a+mIib0wrcyEU1cKOdXA+8JCKlMtufDXVkTBqqultE3gBuxw0dE4wlqWO9icha4HOvfCVwUcB676lqCrBaRP4AzgIuBeoEnJ2VxP1HPgR8p6rrgoyhEjDVSxQFgcDtZqpqal0uADp4df1URHZ45a1wgwEv8YZ2KwKk1wyT0XpNgXhV3ep9DlNxfzWb2JEP2On94ZJWAm7iyMPAOhH5HffvfElmOzPGHO8Z3JlNsYCyZLz/MyKSD5cEUh0MeJ0S8D6Ff/8hmHZsMcXN+jog4DpSVVVNTXB7sxHzeNzsx+cAtwKFA5YFsx8BXg+I47+qOuwE1jMxxpsKZJ2IXAtusFsRqestnoE7e8JrUj4T+COz/VmCMiYdqpoIvIdLUqn+5Nh0I+3wBmLNpmtFJJ93Xaoa8BtusN7bxE2rgIicKSLFMttJBkpybEqJ7pms9w3QyTvWpUBpr3wO0FFETvGWlRGR071lh1Pjy2S9xcCFIlLWW/faHNTB5CLiBiheCPxXRBJEpBdwA9BLRH4EVnFslu7PgO0i8jMwF7hbVTMddd2a+IzJ2Gigf8D7l4D/ef/xPiV7Zzep/gK+A04C+qjqARF5GXdt6ntxbWZbgatzsO9huLb/HcBXQNUM1nsEeEdEuuJ+XDYDe1R1m4g8CHzunSEeBvoB63EXvleIyPfedajj1lPVReK65C8EdgI/5KAOJhfxrmmm57gOEN4I64O8R1BsNHNjYoyIFAKOeNOtnAu8kME1A2N8ZWdQxsSe04D3vLOfQ7j7U4yJOnYGZYwxJipZJwljjDFRyRKUMcaYqGQJyhhjTFSyBGWMMSYqWYIyxhgTlf4ftSK87gMb39gAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "perf_h2o.plot_uplift(metric=\"qini\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAABKoklEQVR4nO3deZyN5fvA8c9lbNmGkITwLS3InuWrrIU2Id+KvraEqCRp7xslpX6WtCpSVLIUFRGRkQpZkpKKpKhRjHXsY67fH/cznMYsZ5Yzz8w51/v1el7nnPvZrvsMc81zP/dz36KqGGOMMblNPr8DMMYYY1JiCcoYY0yuZAnKGGNMrmQJyhhjTK5kCcoYY0yuZAnKGGNMrmQJykQcERkmIm/7HUcwRKSniHwR8DleRP7lvT9DROaIyD4RmelflLmfiFwuIj/5HYfJGEtQJseISFcRWe39ko0VkfkicpnfcaVFRN4UkSezsH+MiNyWrKyFiGzPzPFUtZiqbvE+dgbKAaVV9T/BxCrOQBH5XkQOish2EZkpIpdkJp68QlWXqeqFfsdhMsYSlMkRIjIYeA54CvdL9VzgZeB6H8PK6yoDP6tqQgb2GQfcDQwEzgQuAD4ArsnoyUUkf0b3MSZDVNUWW0K6ANFAPPCfNLZ5E3gy4HMLYHvA563AfcB64CDwOi7RzQcOAIuAUintG7D/Fd77YcDbAetmAjuAfcDnQA2vvC9wHDjmxT/HKz8HeB/YCfwKDEyjXjHAbcnKktdNcQljC7AL+D8gn7euJ/BFsm3PBx734jruxdYvpViTnbcacAJoGGy8qZz/DmCTV/dXgFHJjvEhMDij31UQ/46igNHed/QrcKcXT35vfS9go/fvYQvQL51/T0O8f0/7gOlAYb//r9jyz8WuoExOaAIUBmZn8Tg3AFfi/uq/DpecHgbK4loDBmbyuPNxv7zPAtYC7wCo6mve+2fVNa1dJyL5gDnAt0AFoDUwSETaZrZSno5AA6Ae7qry1rQ2VtWhuKvR6V5sryaPNYXdWuN+SX+dxVg7AI2A6sC7wE0iIgAiUgpoA0wLwXfVB7gKqIP7njokW/83cC1QApesxopIvTSOdyPQDqgK1MIlY5OLWIIyOaE0sEsz1hSVkhdU9S9V/QNYBqxU1W9U9Qgu+dXNzEFVdZKqHlDVo7irq9oiEp3K5pcCZVX1CVU9pu5+0ATg5sycO8AzqrpbVX/HNYV2yeLxUlIaiM2G4zztxXoY93NQ4HJvXWdguar+SfZ/VzcC41R1u6ruAUYGrlTVj1X1F3WWAgsD4krJ86r6p6ruxiXSOpmMy4SItSGbnBAHlBGR/FlMUn8FvD+cwudiGT2giEQBI4D/4K7EEr1VZXBNP8lVBs4Rkb0BZVG4X9QpSQAKJCsrgGuOC7Qt4P1vuKax7BYHlM+G45yMVVVVRKbhEurnQFcgqYdk0N+ViJwL/BBw3JR+lufwz+8p8D0ichUwFHeFnQ8oAnyXRj12BLw/RGi+c5MFdgVlcsJy4CinN8kEOoj7hZLk7Cyc7x/H8pJQ2VS27YprUrsCd6+sStJu3mvy4f63Ab+qasmApbiqXp3K8X8POGaSqrgkFKhSwPtzgT9TOV5a0puaYDFQUUQapLFNMD+H5Od5F+gsIpVxTX/ve+VBf1eq+rvXNFksleQE7uqvYsDnk9+ZiBTyzjsKKKeqJYF5nPo5mjzIEpQJOVXdBzwGvCQiHUSkiIgUEJGrRORZb7N1wNUicqaInA0MysIpfwYKi8g1IlIAeBQolMq2xXHJMw73i/mpZOv/Av4V8Plr4ICIPOA9hxQlIjVF5NJUjj8d6CUiDb0u3hcA9wDTkm13n4iUEpFKuF5204Osa1qx/oOqbsL1nHzX6+peUEQKi8jNIvKgt9k6oJP3Mzof6J3eSVX1G1zHhYnAAlXd663K6HeVnhnA3SJSQURKAg8ErCuI+xnvBBK8q6k2mTyPySUsQZkcoaqjgcG4ZLET99f1nbguzgBv4W6mb8XdO8jML+ikc+0DBuB+Yf6BuypI7bmjKbirmT9wTUwrkq1/HaguIntF5ANVPYG7EV8H15Ms6RdzivesVHUB8CDwBq7JcB4wGXgt2aYfAmtwCeJj77wZ9Y9YU9lmIPAi8BKwF/gF10Fjjrd+LK4n4F9enO8Eee6puKvQqUkFGf2ugjAB929jPfAN7rtMAE6o6gGvbjOAPbgr448yeR6TS4iqTVhojJ9ERIFqqrrZ71jyEu8qabyqVvY7FhMadgVljMkTvGbCq0Ukv4hUwHWIyOqjCyYXswRljMkrBPeA8h5cE99G3L1NE6asic8YY0yuZFdQxhhjciVLUMYYY3IlS1AmrHnTXezxHuQMLEt1CgwRqSIimny07uTTWYhIeRF53Zs65ICI/Cgij4tI0QzEd5aIvCsif3rzOn0pIo2SxZXoTVGStPQIWB+fbDkhIi94625Jtu6QV6/63vpCIjJeRP4Skd3i5paqkEasr4nIT148PZOt6+mdO/B8LQLW1xGRZV4dt4vI/5Ltf5uIbPb2+0REbFQHYwnKhC8RqYIbi02B9tl87DNxI2ScATRR1eK4gWxLAudl4FDFgFVAfdz0F5OBj0UkcDSFPwNHWVDVyUkrko2+cDZuyKeZ3rp3kq0fgBvle623+924gXxr4Yb52QO8kEas33rHWJvK+uXJ4owJWDcVNxTSmUBzYICItAeXhHEPSF/vrf8VNzqFiXCWoEw464578PZNoEfam2bYYNy0Dv9V1a0AqrpNVe9W1fXBHkRVt6jqGFWNVdUT3gjqBYHMTK53A25E79TGBewBTNFTPaOq4kZ++MsbcHc6UCONWF9S1cXAkUzEVgV4x6vjL8AXAee6FpipqhtU9RgwHGgmIhlJ9CYMWYIy4aw7biSEd4C2IlIuG499BTBLVRNT20BE1nujOqS0vJzKPnVwCSrwod2zvGa4X0VkbBpNiMkTUOBxKwPNcCNnJHkdaCoi54hIEeAW3NQjmVVXRHaJyM8i8r9kTaTPAd3FDXF1Ie7KbVFgiCm8r5mFWEwYsARlwpK4qeQrAzNUdQ1uSJ+u2XiKdKeuUNVayQZKDVwGpBBzCdyQT497wzUB/IgbKqg80ArXFDgmhX0r45rOJidf5+kOLFPVXwPKNuGGnPoD2A9cDDyRVp3S8DkuoZyFu5LrgptgMslc3FQch706va6qq7x1nwA3ikgtETkD92yT8s9Ba00EsgRlwlUPYKGq7vI+T+VUM196U2AkBJSltk12TV0BuFEScOPhrVDVp5PKVXWHqv6gqolecrkflwCS64ab+fbXFNaBS1DJk9dLuAFWSwNFgVlk8grKa6r81YvzO1yi6+zV7UxcEnoCN3FlJdwV7QBv30W4USHex43FuBXXfJra+IkmQliCMmHH+2V/I9BcRHaIyA7cCOK1RaQ26U+BEYtLRGltswjoKG7W2NTi2JBCL7ukZXzAdoVwg+Zux03dnhYl5f+3KSWgpOM3xXWCeC/ZqjrAm97kg0dxHSQaikiZdGIIhnKqqe5fuAFdp6hqgqpux43mfnLaDe/+VjVVLYdLVPmB77MhDpOHWYIy4agDcAI3JXkdb7kY13mgO+lMgeGNwv0+MEJESnv3Tbp4x0u6whiDm1p8ste8hrhpIMaISC3vODWS9WoLXG739imASxyHgR7J72mJSEsRqezFWQk3i+yHybb5N25K9ZmpfB89gPe9Eb8DrcLdF4r24hiA6zG467QjuPMUFJHCuMRTQNxUHfm8dVcl3eMTkYuA/wXE+bMrlq4ikk/cdCo34UYlxztOTa+O5+JGeh+nbtZcE8lU1RZbwmrBNSeNTqH8RtwsqvmBW4ENuHsvm3FTYuQL2LYUp6br2AN8CTRNdrxzgEneMQ/g7q0MBYpkINbmuKuNQ0B8wHK5t36wF8Mh3P2i54HiyY7xKvBWKscvjJtWo3UK60rjOpD87W3zBdAwYP184OGAzzFerIFLC2/dKNwUHQdxXdmfAAoE7NsKlxD3ed/XhKTvCdc1f7237w7gaSDK739Htvi/2Fh8xhhjciVr4jPGGJMrWYIyxhiTK1mCMsYYkytZgjLGGJMr5U9/k/BSpkwZrVKlSqb3P3jwIEWLBj1YdZ5l9QwvkVJPiJy6hlM916xZs0tVy562IlTdA3HdW7/GjYC8ATd8C7iHHVfiuvZOBwp65YW8z5u99VUCjvWQV/4T0DagvJ1Xthl4MJi46tevr1mxZMmSLO2fV1g9w0uk1FM1cuoaTvUEVmsKv69D2cR3FGilqrVxD0q2E5HGwDPAWFU9H/d8SW9v+97AHq98rLcdIlIduBk38nE74GURiRKRKNxQLVfhHqDs4m1rjDEmDIQsQXmJMd77WMBbFPfAXtKQK5NxT/2DmwsmaaiW94DWIiJe+TRVPapunLHNQENv2axuDLBjuFEArg9VfYwxxuSskN6D8q5y1gDn4652fgH2qmrSYJzbcUO04L1uA1DVBBHZh3vSvQJuTh9S2GdbsvJGpEBE+gJ9AcqVK0dMTEym6xQfH5+l/fMKq2d4iZR6QuTUNRLqGdIEpW5MszoiUhKYDVwUyvOlEcdruPG9aNCggbZo0SLTx4qJiSEr++cVVs/wEin1hMipayTUM0e6mavqXmAJbpKykgETmVXEjTOG91oJwFsfjZvS4GR5sn1SKzfGGBMGQpagRKSsd+WUNP3BlcBGXKLq7G3Wg1MjHn/Eqfl6OgOfeb07PgJuFpFCIlIVqIbrHbgKqCYiVUWkIK4jxUehqo8xxpicFcomvvK4qQiicIlwhqrOFZEfgGki8iTwDW7aabzXt0RkM7Abl3BQ1Q0iMgP4ATeR3B1e0yEiciewAIgCJqnqhhDWxxhjjEfVLflC2A4XsgSlquuBuimUb8H1wEtefgT4TyrHGgGMSKF8HjAvy8EaY0wESkyE+HjYv98tBw6cep/e588/hzJl4OuvoWrV0MQXcSNJGGNMXqYKR4/Cnj0F2Lw5uMSS2jYHkk9hmYrChaFECShe3L2WKOHKd+3aTZ8+Rfj008KIpH2MzLAEZYwxOeDEiYxdoaS1zfHjAE3TPF++fKeSSVJyKVUKKlc+PdkEbpP8c/HiULBg8rqcICoqinHjSvDww4v4+ed2XHhh9n9nlqCMMSYVqnD4cHBJI71tDh0K7pxFipyeMKpWPb1sx46fadDgglQTS5EiZPtVzbFjx3j55Zd55ZVXWLlyJQMHluQ//2nLOedk73mSWIIyxoSd48eDvxpJb5vExPTPlz//6VcjZcvCeeelfXWSfJ9ixdyxghET8yctWlyQtS8qSKrKnDlzGDJkCJs2baJNmzbs37+fkiVLcs45IWjb81iCMsbkComJcPBg1hPL3r2Xc+xYcOcsVuz0JHH22eknkuRlhQpl/9VKbnHw4EHat2/PZ599xkUXXcTHH3/MVVddheRAhS1BGWOy5OjRrN9TSXp1kxSkrVCh0xNEhQqnyvbu/YOaNc9NN7EUKxbaLtJ53eHDhznjjDMoWrQoFSpU4IUXXqBfv34UKFAgx2KwBGVMBDpxImPdi9PaJpirFZHTE0R0NFSqFFzTV1JZ8eIuQaUlJmYLLVqcmz1fVAQ6cuQIY8eOZcyYMaxcuZJ//etfTJkyxZdYLEEZk0eowpEjmWv6+uOPeqie+hwfn/75AM444/SkkVovsLQSS9Gi4dsEFi5UlRkzZvDAAw/w22+/cf3115PP50tMS1DG5BIJCbBqFXz6KXz/Pfz55+mJJiEh/eNERaWUNI5TpUrGEkvx4pCDrTnGRwkJCbRu3ZrPP/+c2rVrM2nSJFq1auV3WJagjPHTb7/BwoWwYAEsXgx797orjfPPh4oVM94LrHhxd9WT/GolJua7sB/52mRcXFwcpUuXJn/+/LRq1Yru3bvTs2dPoqKi/A4NsARlTI6Kj4elS11CWrgQfvrJlVesCDfcAG3bQuvWcOaZ/sZpwlt8fDzPPvsso0aNYsGCBVx++eUMHTrU77BOYwnKmBBKTIRvvz2VkL74wj2jc8YZ0KIF3H67S0oXXWT3aEzoJSYmMmXKFB5++GFiY2O5+eabqVy5st9hpcoSlDHZbMcOdx9pwQL3+vffrrx2bRg0yCWkpk3d+GbG5BRV5corr+Szzz6jUaNGvP/++zRp0sTvsNJkCcqYLDpyBL788tRV0rffuvKyZaFNG5eQrrgCypf3N04TmX777TcqVapEvnz5+O9//0vv3r25+eabfe+hFwxLUMZkkCr8+OOphBQT48ZrK1AALrsMRo50ial2bXsQ1Phn3759PPnkkzz//PNMnDiRbt260atXL7/DyhBLUMYEYfduWLTIJaSFC2HbNld+4YVw223uKql5czc6gTF+SkhIYMKECTz22GPExcXRs2dPWrdu7XdYmWIJypgUHD8OK1ee6gK+apW7coqOds11//ufu0rKxfeXTYTq1KkTc+bMoXnz5owZM4Z69er5HVKmWYIyxrNly6mE9Nln7uHYfPmgUSMYOtQlpEsvDX60aWNyysaNG6lcuTJFihThzjvvpFevXnTo0CFHBnQNJfuvZiLWgQPw5ZelmTnTJabNm135uefCTTedeiapZElfwzQmVbt27WLYsGGMHz+exx9/nEceeYQ2bdr4HVa2sQRlIkZiIqxde6pzw1dfQULCJRQpAi1bwsCB7irpggvsmSSTux07dowZM2bQoUMH4uPj6devH3379vU7rGxnCcqEtT/+ONWx4dNPIS7OldetC0OGwFlnrWPAgDrpjpBtTG7Su3dv3n77bdq1a8fo0aOpXr263yGFhCUoE1YOH4Zly9xV0oIFsGGDKz/7bLjmGneFdOWVcNZZrjwmZq8lJ5MnfPPNN5x99tmUL1+eIUOGcMkll3D//ff7HVZIWYIyeZqqG/k76Srp88/dg7MFC0KzZtCjh7uXdMkl1mxn8qbY2FgeeeQR3nzzTfr3789LL71E7dq12bNnj9+hhVzIEpSIVAKmAOUABV5T1XEiMgzoA+z0Nn1YVed5+zwE9AZOAANVdYFX3g4YB0QBE1V1pFdeFZgGlAbWAN1UNcjJnk1etWuXa65LSkp//unKL77YjW3Xpo17JqlIEX/jNCYrDh8+zOjRoxk5ciTHjh1j8ODBPProo36HlaNCeQWVANyrqmtFpDiwRkQ+9daNVdVRgRuLSHXgZqAGcA6wSEQu8Fa/BFwJbAdWichHqvoD8Ix3rGkiMh6X3F4JYZ2MD44dg+XLT3UBX7vWXTmVKuWa69q0cUulSn5Hakz2eeihhxg3bhwdO3bk2Wef5fzzz/c7pBwXsgSlqrFArPf+gIhsBCqkscv1wDRVPQr8KiKbgYbeus2qugVARKYB13vHawV09baZDAzDElSep+q6fCclpCVL3DQVUVHQpAk8/rhrtqtf35UZEy6WL19O8eLFqVmzJvfddx8dOnSI6Hm8RFVDfxKRKsDnQE1gMNAT2A+sxl1l7RGRF4EVqvq2t8/rwHzvEO1U9TavvBvQCJeMVqjq+V55JWC+qtZM4fx9gb4A5cqVqz9t2rRM1yU+Pp5iETCeTU7XMz4+im++KcWqVWeyenUpYmPPAKB8+cNceuluLr10D3Xq7KFYsRPZfF77eYabvFjXHTt28Nprr7FkyRKaN2/OsGHD0t0nL9YzNS1btlyjqg1OW6GqIV2AYrj7Q528z+Vw95LyASOASV75i8B/A/Z7HejsLRMDyrt525bBXVkllVcCvk8vnvr162tWLFmyJEv75xWhrmdCguqKFaqPP67atKlqVJQqqBYrptq+vepLL6lu2hTSEFTVfp7hKC/Vdf/+/frQQw9poUKFtHDhwvq///1PDxw4ENS+eame6QFWawq/r0Pai09ECgDvA++o6iwvIf4VsH4CMNf7+IeXZJJU9MpIpTwOKCki+VU1Idn2JpfYvfvU7LDbtp16SHbRItizx/Wsq18fHnjANds1bux64BkTCV566SWefvppbrnlFp5++mkq2Y3UfwhlLz7BXQVtVNUxAeXl1d2fAugIfO+9/wiYKiJjcJ0kqgFfAwJU83rs/YHrSNFVVVVEluCusKYBPYAPQ1Ufk3FPPOHGsOvRA9ascd3BAc45B66//tQ8SWXK+BunMTlpyZIlqCqtWrXirrvuomXLljRq1MjvsHKlUF5BNcU1x30nIuu8soeBLiJSB9f1fCvQD0BVN4jIDOAHXA/AO1T1BICI3AkswDUNTlJV7/FLHgCmiciTwDe4hGhygZEjXXICmDwZatSAUaNcUqpRw55JMpFn06ZNDBkyhI8++ojWrVvTqlUrihYtaskpDaHsxfcF7uonuXlp7DMCd18qefm8lPZT17OvYfJy469nn4WHHoKuXV1yio9301RYUjKRaM+ePTzxxBO8+OKLFC5cmKeeeop77rnH77DyBBtJwmSr0aPd/aSbb3bJKX9+Gw3cRLaPP/6YcePG0bt3b4YPH87ZZ5/td0h5hiUok23GjnUDsN54I7z1ls2bZCKTqjJ//nz27NnDLbfcQteuXalXr17YDugaSvn8DsCEh3HjYPBg6NwZ3n7bkpOJTN9//z1t27blmmuu4fnnn0dVyZcvnyWnTLIEZbLsxRdh0CDo1AmmToUCBfyOyJictXPnTvr370/t2rVZtWoVY8eOZdmyZXl+Rlu/2d+5Jktefhnuugs6dIB337XkZCLTxo0bmThxInfccQdDhw6ldOnSfocUFixBmUwbPx7uuAPat4fp0+0BWxM5VJVZs2axadMmHnzwQZo1a8bWrVupUCGt4UZNRlkTn8mU116D/v3h2mthxgxLTiZyrFmzhubNm9O5c2dmzJjB8ePHASw5hYAlKJNhr78O/frB1VfDe+9hM9KaiLBjxw569OhBgwYN+PHHHxk/fjxff/01BaxdO2Ssic9kyBtvQJ8+0K4dvP++JScTOQ4ePMjs2bN54IEHeOihh4iOjvY7pLBnCcoEbfJk6N3bTRI4ezYULux3RMaETmJiIu+88w5ffPEFr776Kueddx7bt2+nRIkSfocWMayJzwTlrbegVy9o3Ro++MCSkwlvX375JY0bN6Z79+6sXbuWAwcOAFhyymGWoEy63nkHevaEli3hww/hjDP8jsiY0NixYwc33ngjl112GX/++SdTpkxh5cqVFC9e3O/QIpI18Zk0vfsudO8OzZrBnDlQpIjfERkTOoULF2blypUMGzaMIUOGULRoUb9DimiWoEyqpk+H//4XLrsM5s615GTCz4kTJ3j99dd57733mD9/PiVLlmTTpk0UtOcmcgVr4jMpiokpyy23QNOm8PHHYH9ImnCzaNEi6tatS79+/Th8+DC7du0CsOSUi1iCMqd5/30YPrw6jRu75FSsmN8RGZN9du3axXXXXceVV15JfHw8M2fO5PPPP6dcuXJ+h2aSsSY+8w+zZ7u5nC6+eD/z50dj94ZNuEhMTCRfvnxER0ezc+dOnnnmGQYOHEhh65Kaa9kVlDnpww/dXE4NGsAzz6y35GTCwvHjxxk3bhw1a9bkwIEDFChQgOXLl3P//fdbcsrlLEEZwPXQ+89/oF49+OQTKFr0hN8hGZMlqsqcOXOoWbMmgwYNomLFiuzduxfApsHIIyxBGT7+GG64AWrXhgULwEZwMXldfHw8V155Je3bt0dEmDt3LgsWLKBSpUp+h2YywO5BRbj5891Eg5dcAgsXQsmSfkdkTOYdOXIEgKJFi1K2bFmef/55br/9dhvQNY+yK6gItmABdOwINWrAp59CqVJ+R2RM5hw5coSnn36ac889lx07diAivPvuu9x1112WnPKwdBOUiHQSkU0isk9E9ovIARHZnxPBmdBZuBCuvx4uusglpzPP9DsiYzJOVZk+fToXXXQRDz/8ME2aNPE7JJONgrmCehZor6rRqlpCVYurarojJopIJRFZIiI/iMgGEbnbKz9TRD71kt6nIlLKKxcReV5ENovIehGpF3CsHt72m0SkR0B5fRH5ztvnebE7n0FZvNglpwsvhEWLwGanNnlRQkICLVq04OabbyY6OprFixfz4YcfcvbZZ/sdmskmwSSov1R1YyaOnQDcq6rVgcbAHSJSHXgQWKyq1YDF3meAq4Bq3tIXeAVcQgOGAo2AhsDQpKTmbdMnYL92mYgzosTFua7k553nklOZMn5HZEzG7NmzB4D8+fNz2WWXMWHCBNauXUurVq18jsxkt2AS1GoRmS4iXbzmvk4i0im9nVQ1VlXXeu8PABuBCsD1wGRvs8lAB+/99cAUdVYAJUWkPNAW+FRVd6vqHuBToJ23roSqrlBVBaYEHMuk4tFHYd8+Nwhs2bJ+R2NM8OLj43nssceoWLEiK1asAGDEiBHcdtttREVF+RydCYVgevGVAA4BbQLKFJgV7ElEpApQF1gJlFPVWG/VDiBpfJEKwLaA3bZ7ZWmVb0+hPKXz98VdlVGuXDliYmKCDf008fHxWdrfTz//XIxXX61Pp05/EBe3mbSqkZfrmRFWz9wvMTGRhQsXMnHiROLi4mjVqhW//fbbyR57yeXlumZEJNQz3QSlqr2ycgIRKQa8DwxS1f2Bt4lUVUVEs3L8YKjqa8BrAA0aNNAWLVpk+lgxMTFkZX+/qLqrp7JlYeLEipQsWTHN7fNqPTPK6pm7qSqtWrUiJiaGRo0aMWfOnHQ7QuTVumZUJNQz1QQlIver6rMi8gLuiukfVHVgegcXkQK45PSOqiZdcf0lIuVVNdZrpvvbK/8DCHyKrqJX9gfQIll5jFdeMYXtTQoWLoQvv4SXXrJnnUzu9/vvv1OpUiVEhC5dunDbbbfRpUsX8uWzJ2MiSVo/7aSOEauBNSksafJ61L0ObFTVMQGrPgKSeuL1AD4MKO/u9eZrDOzzmgIXAG1EpJTXOaINsMBbt19EGnvn6h5wLBPgp5+gc2eoWhVuvdXvaIxJ3b59+7jvvvuoVq0a06dPB6Bv377ccsstlpwiUKpXUKo6x3udnNo26WgKdAO+E5F1XtnDwEhghoj0Bn4DbvTWzQOuBjbj7nn18s6/W0SGA6u87Z5Q1d3e+wHAm8AZwHxvMcncfjvkzw+ffQY2NqbJjRISEpgwYQKPPfYYcXFx9OrVi+bNm/sdlvFZuvegRKQs8ABQHTj5601V0+zTqapfAKk9l9Q6he0VuCOVY00CJqVQvhqomVYcke677yAmBkaNgipV/I7GmJR16NCBjz/+mBYtWjBmzBjq1q3rd0gmFwjmmvkdXHNfVeBxYCunrmZMLjdqlJuqvUeP9Lc1Jidt3LiRw4cPAzBgwABmz57NZ599ZsnJnBRMgiqtqq8Dx1V1qareCtgTcXnA0qXwzjvQt689kGtyj127dnHnnXdyySWX8PzzzwNw9dVX06FDB5sGw/xDMM9BHfdeY0XkGuBPwEZuy+U2b4abbnIjRjz+uN/RGAPHjh3jxRdf5IknniA+Pp5+/fpxq/XaMWkIJkE9KSLRwL3AC7gHd+8JaVQmy26/HY4fh/fegxLpjpxoTOj17NmTd999l3bt2jF69GiqV6/ud0gmlwvmQd253tt9QMvQhmOyw5YtbkDY4cPdPE/G+GXdunWUL1+ecuXKMWTIELp37067djZkpglOML34nk+heB+wWlXtuaNc6PXXIV8+6NnT70hMpIqNjeXRRx/ljTfe4K677mLcuHHUq1cv/R2NCRBMJ4nCQB1gk7fUwo3a0FtEngtZZCZTDhyACRPg2muhYtqjGRmT7Q4fPsyIESOoVq0ab731FoMHD+ZxuwlqMimYe1C1gKaqegJARF4BlgGXAd+FMDaTCffe66bUePhhvyMxkej+++/nxRdfpFOnTjzzzDOcf/75fodk8rBgElQpoBiuWQ+gKHCmqp4QkaMhi8xk2KFDMHWqG86oUSO/ozGRYsWKFURHR3PxxRdz//3307lzZxsFwmSLYGfUXScib4jIm8A3wP+JSFFgUSiDMxnz0Udw8CB06+Z3JCYS/Pbbb3Tp0oUmTZrwxBNPAFCpUiVLTibbBNOL73URmYebzRbgYVX903t/X8giMxly/Dg89xxER0PTpn5HY8LZgQMHGDlyJGPGuDGgH330UR544AGfozLhKJgmPryRw63HXi514oTrsbdyJYwZAza5qAmlcePG8dRTT3HLLbfw1FNPce655/odkglTQSUok3sdPw69erl7T08/DffYI9QmBJYsWUJUVBTNmjXj7rvv5sorr6SR3eg0IWYTrORhhw9Dx45uvL2nnoIHH/Q7IhNuNm3aRIcOHWjVqhUjRowAoHjx4pacTI4IKkGJyGUi0st7X1ZEqoY2LJOeffugbVuYNw9eeQUeesjviEw42bNnD4MHD6ZGjRosXryYESNG8MEHH/gdlokwwYwkMRRoAFwIvAEUAN7GTUhofHLDDbB8Obz7rhsU1pjsNHv2bJ577jl69+7N8OHDOfvss/0OyUSgYO5BdQTqAmsBVPVPESke0qhMmn7+2Y2199RTlpxM9lBV5s+fz4EDB7jpppvo0aMHjRo1okaNGn6HZiJYME18x7zZbhXAe/7J+Gj6dPfaubO/cZjw8P3339OuXTuuueYannvuOVSVqKgoS07Gd8EkqBki8ipQUkT64B7OnRDasExqjh1zs+Q2bw42iozJip07d9K/f39q167N119/zdixY1m6dKlNGmhyjWAe1B0lIlcC+3H3oR5T1U9DHplJ0fDhsH+/6xRhv0dMVqxfv56JEydyxx13MHToUEqXLu13SMb8QzCdJAYD0y0p+W/lSnffqWdP14PPmIxQVWbNmsWWLVu47777aN26NVu2bKFSpUp+h2ZMioJp4isOLBSRZSJyp4iUC3VQ5nSHDkH37m4Kjeee8zsak9f89NNPNG/enM6dOzNt2jQSEhIALDmZXC3dBKWqj6tqDeAOoDywVERskNgc9uCDrvfeG2+48faMCUZsbCw9evTg9ttv58cff2T8+PGsXLmS/PltEBmT+2VkJIm/gR1AHHBWehuLyCQR+VtEvg8oGyYif4jIOm+5OmDdQyKyWUR+EpG2AeXtvLLNIvJgQHlVEVnplU8XkYIZqEuesngxvPACDBwIrVr5HY3JS/bv38/777/PzTffzKZNm+jXr58lJ5NnpJugRGSAiMQAi4HSQB9VrRXEsd8E2qVQPlZV63jLPO8c1YGbgRrePi+LSJSIRAEvAVcB1YEu3rYAz3jHOh/YA/QOIqY8Z+9ed8/pwgvdWHvGpCUxMZG3336bO+64A4ALL7yQ7du3069fP6Lt0tvkMcH8KVUJGKSq6zJyYFX9XESqBLn59cA0VT0K/Coimzk1vcdmVd0CICLTgOtFZCPQCujqbTMZGAa8kpEY84K774bYWPjqKyhSxO9oTG725Zdfcs8997Bq1SoaNGhAfHw8xYoVo2TJkn6HZkympJqgRKSEqu4H/s/7fGbgelXdnclz3iki3YHVwL2qugeoAKwI2Ga7VwawLVl5I9yV3F5VTUhh+5Tq0hfoC1CuXDliYmIyGTrEx8dnaf+MWLasDFOm1KRbt60cOrSVHDotkLP19FM41DMuLo4XXniBpUuXUqZMGR588EGuvPJKVq9efXKbcKhnsCKlrhFRT1VNcQHmeq+/Alu816RlS2r7JTtGFeD7gM/lgChc0+IIYJJX/iLw34DtXgc6e8vEgPJu3rZlcFdWSeWVAs+T1lK/fn3NiiVLlmRp/2D99Zdq2bKq9eqpHj2aI6f8h5yqp9/CoZ67du3SChUq6NChQzU+Pj7FbcKhnsGKlLqGUz2B1ZrC7+tUr6BU9VrvNdtGLlfVv5Lei8gEYK738Q8vySSp6JWRSnkcbmSL/OquogK3z/NUoW9f90DulClQMGy7f5jMOHHiBJMmTWLWrFnMnTuX0qVL88svv1CoUCG/QzMmWwXTSWJxMGXBEJHyAR87Akk9/D4CbhaRQt5UHtWAr4FVQDWvx15BXEeKj7yMuwR3hQXQgzCa8XfyZPjwQxgxAmw4NBNo0aJF1K1bl759+xIfH8/u3a6l3ZKTCUdp3YMqDBQByohIKSBpYJ0SpHG/J2D/d4EW3v7bgaFACxGpgxt4divQD0BVN4jIDOAHIAG4Q1VPeMe5E1iAaxqcpKobvFM8AEwTkSeBb3DNgnneoUMwaBA0a+ZejQE3bt6tt97K3LlzqVq1KjNnzuSGG26wcfNMWEurF18/YBBwDrCGUwlqP+4+UJpUtUsKxakmEVUdgbsvlbx8HjAvhfItnOrpFzaWL3eTEd53H0RF+R2N8VtiYiL58uUjOjqaHTt28MwzzzBw4EAKFy7sd2jGhFxa96DGAeNE5C5VfSEHY4pY27fDI4/AGWdAixZ+R2P8dPz4cV5++WUmTpzIihUrKFq0KCtXriRfvow8W29M3hbMaOYviEhN3IOyhQPKp4QysEjUrBn8+isMGwbFivkdjfGDqjJ37lyGDBnCzz//zBVXXMGePXsoWrSoJScTcYKd8r0FLkHNw43q8AVgCSobHTvmklObNjB0qN/RGD/s37+fTp06sXjxYi688ELmzJnDNddcY/eZTMQK5k+yzkBrYIeq9gJqAzZmSjb7/Xf32rVr2tuZ8HP06FEAihcvTqlSpRg3bhzfffcd1157rSUnE9GCSVCHVTURSBCRErhBY22M/my2fLl7/de//I3D5JwjR44wcuRIKleuzPbt2xERZs6cycCBAylQoIDf4Rnju2DG4lstIiVx07yvAeKB5aEMKhJ99517rV3b3zhM6KkqM2fO5IEHHmDr1q20b9+eEydO+B2WMblOMJ0kBnhvx4vIJ0AJVV0f2rAiz6JFrpNEiRJ+R2JC6fjx47Ru3Zply5ZRq1YtFi1aROvWrf0Oy5hcKa0HdeultU5V14YmpMizciV88w2MGeN3JCZU9u7dS8mSJSlQoABNmjShe/fu9OrViyh72M2YVKV1BTU6jXWKm+7CZINhw6BMGejTx+9ITHY7ePAgzz77LKNHjyYmJoYGDRrwzDPP+B2WMXlCWg/qtszJQCLV8uXwySfwzDP27FM4SUxM5K233uLhhx/mzz//5KabbuKss9KdiNoYEyCY56C6p1RuD+pmj6SrpwED0t3U5BGqSqtWrVi6dCkNGzZk5syZ/Pvf//Y7LGPynGB68V0a8L4w7pmotdiDuln21VewcCE8+6xdPYWDbdu2UbFiRUSEm266iT59+tClSxcbAcKYTAqmF99dgZ+9LufTQhVQJBk+HMqWtaunvG7fvn2MGDGCcePGMXXqVG644Qb69+/vd1jG5HmZ+dPuIJBtkxhGqmPHYPFi6N4dihb1OxqTGQkJCYwfP55q1aoxatQounbtSpMmTfwOy5iwEcw9qDm4XnvgElp1YEYog4oEGzfC8eNQv77fkZjMuu666/jkk09o1qwZY8eOpV69VJ/MMMZkQjD3oEYFvE8AflPV7SGKJ2Is9uYktnvnecuPP/5I1apVKVSoEP3796dPnz507NjRxswzJgTSbeJT1aWquhQ3a+1G4JCInBnyyMKYKsyZA1WqQOXKfkdjgrFr1y7uvPNOatasyUsvvQRA+/bt6dSpkyUnY0IkmCa+vsATwBEgETezrgI2rGkmrV8PMTHw1FN+R2LSc+zYMV588UWGDx/O/v376devH926dfM7LGMiQjBNfPcBNVV1V6iDiRSbNrnXq67yNw6Tvm7dujFjxgzatm3L6NGjqVGjht8hGRMxgunF9wtwKNSBRJJly6BgQTjvPL8jMSlZt24dO3fuBGDIkCHMmzePTz75xJKTMTksmAT1EPCViLwqIs8nLaEOLFydOAFTp0LHjlC8uN/RmECxsbH07t2bevXqMXLkSAAuvfRSrrJLXWN8EUwT36vAZ8B3uHtQJgtWrIBdu1yCMrnD4cOHGTNmDE8//TTHjh1j8ODBPProo36HZUzECyZBFVDVwSGPJEK8+Sbkzw9t2/odiUly77338sorr9CxY0eeffZZzj//fL9DMsYQXBPffBHpKyLlReTMpCW9nURkkoj8LSLfB5SdKSKfisgm77WUVy5e0+FmEVkfOBeViPTwtt8kIj0CyuuLyHfePs9LHujru28fvPUW9O4NJUv6HU1kW7FiBT///DMA999/P0uWLGHWrFmWnIzJRYJJUF3w7kPhpnxfA6wOYr83gXbJyh4EFqtqNWCx9xngKqCat/QFXgGX0IChQCOgITA0Kal52/QJ2C/5uXKdWbPg6FG49Va/I4lcv//++8khiYYPHw5AlSpVaNGihb+BGWNOE8yDulVTWNJ9BkpVPwd2Jyu+HpjsvZ8MdAgon6LOCqCkiJQH2gKfqupuVd0DfAq089aVUNUVqqq4kdU7kMvNmwcVK8Kll6a/rcle8fHxPProo1x44YXMnj2bRx99lFdeecXvsIwxacjp+aDKqWqs934HUM57XwHYFrDddq8srfLtKZSnyHvYuC9AuXLliImJyUToTnx8fKb2P3o0H/Pn/5vmzXeydOlPmT5/TslsPXOrN998k8mTJ9O6dWv69OlDuXLlWL16ddjVMzWRUk+InLpGQj19mw9KVVVENP0ts05VXwNeA2jQoIFmpTknJiYmU81B48bBwYPwwAPladasfKbPn1MyW8/cJCYmhgIFCtC0aVPq1atH//79adSo0Wnb5PV6BiNS6gmRU9dIqGcwTXx3BSx9gHpAZqfX+8trnsN7/dsr/wOoFLBdRa8srfKKKZTnWm+/DQ0bQrNmfkcS/jZv3kzHjh1p2bIlI0aMAKBEiRKnJSdjTO6W0/NBfQQk9cTrAXwYUN7d683XGNjnNQUuANqISCmvc0QbYIG3br+INPZ673UPOFauExcHa9bANdf4HUl427t3L/feey/Vq1dn0aJFjBgxgvfff9/vsIwxmRSy+aBE5F2gBVBGRLbjeuONBGaISG/gN+BGb/N5wNXAZtywSr0AVHW3iAwHVnnbPaGqSR0vBuB6Cp4BzPeWXOnDD90I5jYgQWjNnDmTsWPHcuutt/Lkk09y9tln+x2SMSYLQjYflKp2SWVV6xS2VeCOVI4zCZiUQvlqoGZ6ceQGEydC9eo2OWEozJ8/n4MHD9K5c2d69epF48aNueSSS/wOyxiTDVJt4hOR80WkadJ8UN7yJVBZRGyY0yAdOQKrV8N110G+zDSomhRt2LCBdu3acfXVV/Pcc8+hquTPn9+SkzFhJK1fmc8B+1Mo3++tM0GYN89N7d6qld+RhIedO3fSv39/atWqxcqVKxkzZgyfffaZTRpoTBhKq4mvnKp+l7xQVb8TkSqhCym8vPMOREdbgsoua9euZcKECQwYMIBhw4ZRunRpv0MyxoRIWgmqZBrrzsjmOMLS+vVueKPHHnMDxJqMU1Vmz57N77//zqBBg2jbti1btmzh3HPP9Ts0Y0yIpdXEt1pE+iQvFJHbcOPxmXSMGgVFi8KgQX5HkjetWbOGFi1acMMNN/D2229z4sQJAEtOxkSItP6uHwTMFpFbOJWQGgAFAZvNKB1xce7h3LvvhlKl0t/enBIbG8vDDz/M5MmTKVOmDOPHj6d3795ERUX5HZoxJgelmqBU9S/g3yLSklPduT9W1c9yJLI8btUq9+xT+/Z+R5L37NmzhxkzZnDffffx8MMPEx0d7XdIxhgfpHtnRFWXAEtyIJawsmgRFCxoI5cHIzExkalTp7J69Wqee+45qlevzvbt2ylll57GRDR7MidEvvgCmjSBYpkdtTBCfPXVVzRp0oRu3brx5ZdfcujQIQBLTsYYS1ChcPAgrFtnI0ekJTY2lptuuommTZuyfft2Jk+ezMqVKylSpIjfoRljcgnr/BwCc+a4mXM7dPA7ktwrf/78LFu2jKFDh3LfffdRtGhRv0MyxuQylqBC4IsvXNPev//tdyS5x4kTJ5g0aRJz585l9uzZlClThl9//ZVChQr5HZoxJpeyJr5sduIEfPwxXHYZWK9oZ/HixdSrV4++ffsSFxfH7t27ERFLTsaYNFmCymYTJsDWrXDbbX5H4r+///6b9u3bc8UVV7B//35mzJjBsmXLKFOmjN+hGWPyAGviy0aJiW70iCZNoFMnv6Pxj6oiIkRHR/P777/z9NNPM2jQIAoXLux3aMaYPMQSVDbq1Qt++QUefxwicXDt48eP8/LLL/PGG2/w1VdfUaRIEdauXUs+m2fEGJMJ9psjm6jClCnu/X/+428sOU1VmTt3LjVr1mTQoEGULVuWPXv2AFhyMsZkmv32yCarV7vXN990I0hEin379tGmTRuuu+46RIQ5c+awcOFCKlSo4Hdoxpg8zpr4ssns2a7X3nXX+R1Jzjh27BgFCxakRIkSFC1alHHjxtG/f38KFCjgd2jGmDBhV1DZZNYsaNECzjzT70hC68iRI4wcOZIqVaoQGxuLiPDBBx8wcOBAS07GmGxlCSobbNwIP/0U3j33VJUZM2Zw8cUX89BDD3HppZdy/Phxv8MyxoQxa+LLBrNmuddwHdro2LFjXHHFFSxbtoxatWqxaNEiWrdu7XdYxpgwZwkqG3zwATRqBOec43ck2Wvfvn1ER0dTsGBBLr30Urp168att95qEwcaY3KEL018IrJVRL4TkXUistorO1NEPhWRTd5rKa9cROR5EdksIutFpF7AcXp4228SkR5+1OXPP10Pvuuv9+PsoXHw4EHeeOMNKlasyLp16wAYPXo0ffr0seRkjMkxft6DaqmqdVS1gff5QWCxqlYDFnufAa4CqnlLX+AVcAkNGAo0AhoCQ5OSWk5atcq9tmqV02fOfomJiUyePJkLLriAKVOmcM0113BmuPf6MMbkWrmpk8T1wGTv/WSgQ0D5FHVWACVFpDzQFvhUVXer6h7gU6BdDsfML7+412rVcvrM2SsxMZEWLVrQs2dPKlasyAsvvMC0adM499xz/Q7NGBOh/LoHpcBCEVHgVVV9DSinqrHe+h1AOe99BWBbwL7bvbLUyk8jIn1xV1+UK1eOmJiYTAceHx//j/2XLq1GsWJn8e23X+bJ4Y127txJmTJlEBHq1KnD5ZdfTuvWrTl06FCWvqe8IvnPM1xFSj0hcuoaEfVU1RxfgAre61nAt0AzYG+ybfZ4r3OBywLKFwMNgCHAowHl/wOGpHfu+vXra1YsWbLkH58vu8wtec3evXv1vvvu04IFC+qHH3542vrk9QxXVs/wEyl1Dad6Aqs1hd/XvjTxqeof3uvfwGzcPaS/vKY7vNe/vc3/ACoF7F7RK0utPEdt3Ag1auT0WTMvISGB8ePHU61aNUaNGkXXrl1p0KBB+jsaY0wOy/EmPhEpCuRT1QPe+zbAE8BHQA9gpPf6obfLR8CdIjIN1yFin6rGisgC4KmAjhFtgIdysCrs2gVxcXDRRTl51qy55pprWLhwIc2aNWPs2LHUq1cv/Z2MMWk6fvw427dv58iRIzl2zujoaDZu3Jhj58sOhQsXpmLFikGPOuPHPahywGxxN2zyA1NV9RMRWQXMEJHewG/Ajd7284Crgc3AIaAXgKruFpHhgNePjidUdXfOVcONHgFwwQU5edaM++mnn6hatSoFCxbk9ttvp2/fvnTq1AnJizfNjMmFtm/fTvHixalSpUqO/b86cOAAxYsXz5FzZQdVJS4uju3bt1O1atWg9snxBKWqW4DaKZTHAacNT+C1T96RyrEmAZOyO8Zgbd3qXv/1L78iSNuuXbsYNmwY48ePZ8yYMQwcOJCOHTv6HZYxYefIkSM5mpzyIhGhdOnS7Ny5M+h9bCSJLPj1V/ea23piHzt2jJdeeoknnniC/fv3069fP7p06eJ3WMaENUtO6cvod2QJKgsWL3YdJIoU8TuSf7rlllt47733aNu2LaNHj6ZGXurFYYwxntz0oG6eogpr1rgpNnKDdevWsXu3uwV37733Mm/ePD755BNLTsaYFL355pvceeedAIwfP54p3pTgP/74I3Xq1KFu3br88ssvTJ061bcYLUFl0u7dcOAAnHeev3HExsbSu3dv6tWrxzPPPANA48aNueqqq/wNzBiTZ9x+++10794dgA8++IDOnTvzzTffsG3bNl8TlDXxZdLmze7Vrw4Shw8fZuzYsTz11FMcO3aMe+65hwcffDD9HY0xITVoEHhjLGebOnXguefS3mbr1q1ce+21fP/99wCMGjXq5GgTtWvXZunSpSQkJDBp0iQaNmz4j32HDRtGsWLFqF69Os899xxRUVEsXryYw4cPs3HjRurUqUOPHj245557srdi6bAElUlJ/wBr1fLn/HfffTcTJkygQ4cOPPvss1TL64MBGmNC5tChQ6xbt47PP/+cW2+99WQSS+7qq6/m9ttvp1ixYgwZMoSYmBhGjRrF3LlzczhixxJUJq1bByVKQJUqOXfOlStXUqZMGc477zweeOABunTpQsuWLXMuAGNMutK70vFDUi/eZs2asX//fvbu3etvQEGye1CZ9O237rI7J3qW/v7773Tt2pXGjRszfPhwAM477zxLTsaYk/Lnz09iYuLJz4GjWiTv3p1XusRbgsqEEydg/XqXoEIpPj6eRx99lAsvvJDZs2fzyCOP8MILL4T2pMaYPKlcuXL8/fffxMXFcfTo0X80y02fPh2AL774gujoaKKjo4M6ZvHixTlw4EBI4g2GNfFlwvr1cPAg1K0b2vOMHDmSESNG0LVrV55++mmbm8kYk6oCBQrw2GOP0bBhQypUqMBFAYOEFi5cmLp163L8+HEmTQp+8J1atWoRFRVF7dq16dmzZ453kvBlug0/l+yYbqNXL9UzzlDduTNLh0r1+MuXL1dV1d27d598n9PCaSj/tFg9w48fdf3hhx9y/Jz79+8ParvmzZvrqlWrQhxN8FL6rshN023kdfPmwQ03QJky2XfMzZs306lTJ1q2bMmIESMAKFWqFI0bN86+kxhjTB5iTXwZtHdvAf76K/ua9/bu3cuTTz7J888/T8GCBXnyyScZPHhw9hzcGBPx8vKsu5agMig2tjAA2fXY0dSpUxkzZgy33norw4cPp3z58tlzYGOMyeMsQWVQXFwhACpUyPwx5s+fz9GjR+nQoQN9+vThsssuo5ZfT/waY0wuZfegMmjnzoIAnHNOxvfdsGEDV111FVdffTVjxoxBVSlQoIAlJ2OMSYElqAyKjT2DwoXhrLOC32fXrl3ccccd1K5dmxUrVjB69GgWLVqUZx6WM8YYP1iCyqDffivCRRdBvgx8c8uXL+fVV1+lf//+bN68mcGDB1OwYMHQBWmMMVlUpUoVdu3a5WsMdg8qA1Rh06biXHttetspH3zwAdu3b+euu+7i2muvZdOmTVStWjVnAjXGRLSTzxFl5C/pXMgSVAZs2wZ79hQkrUeT1q5dy+DBg1m6dCkNGjRgwIABREVFWXIyJoK0SGEm0xtvvJEBAwZw6NAhrr766tPW9+zZk549e7Jr1y46d+78j3XBdBXfunUrbdu2pVGjRqxZs4aGDRvy3XffcfjwYTp37szjjz8OuCujHj16MGfOHI4fP87MmTO56KKLiIuLo0uXLvzxxx80adIE9/ysM2bMmJMjUNx2220MGjSIrVu30q5dOxo3bsxXX33FpZdeSq9evRg6dCh///0377zzzmnTemRU3k6vOSxphPratU9fFxsbS69evWjQoAEbNmzglVdeYfny5URFReVskMaYiLVp0yYGDBjAhg0bGD16NKtXr2b9+vUsXbqU9evXn9yuTJkyrF27lv79+zNq1CgAHn/8cS677DI2bNhAx44d+f333wFYs2YNb7zxBitXrmTFihVMmDCBb775BnADDNx77738+OOP/Pjjj0ydOpUvvviCUaNG8dRTT2W5PnYFlQE//OBeq1c/fd3OnTuZPn06Q4YM4ZFHHgl6MEZjTPhJ64qnSJEiaa4vU6ZMph+urVy58snRZ2bMmMFrr71GQkICsbGx/PDDDyd7DHfq1AmA+vXrM2vWLAA+//zzk++vueYaSpUqBbgBZjt27EjRokVP7rts2TLat29P1apVueSSSwCoUaMGrVu3RkS45JJL2Lp1a6bqECjPJygRaQeMA6KAiao6MlTn+vlnKFnyGKVKFSQxMZGpU6eybt06Ro0aRa1atdi2bRulS5cO1emNMSZNSUnk119/ZdSoUaxatYpSpUrRs2fPf0y/UaiQe54zKiqKhISETJ8v6TgA+fLlO/k5X758WTruyWNm+Qg+EpEo4CXgKqA60EVEUri+yR4//wwVKhzmq6++okmTJnTr1o2lS5dy+PBhAEtOxphcYf/+/RQtWpTo6Gj++usv5s+fn+4+zZo1Y+rUqYAbTGDPnj0AXH755XzwwQccOnSIgwcPMnv2bC6//PKQxp8kTycooCGwWVW3qOoxYBpwfahOdsUVf5KQ0JWmTZuybds23nzzTVauXMkZZ5wRqlMaY0yG1a5dm7p163LRRRfRtav7nZWeoUOH8vnnn1OjRg1mzZp1cnqfevXq0bNnTxo2bEijRo247bbbqBvquYaSpDTEeV5ZgM64Zr2kz92AF9PaJyvTbcTGxmqpUqX0scce0/j4+EwfJy+IlOkZrJ7hx6bbyN0yMt2GaEBXwrxGRDoD7VT1Nu9zN6CRqt6ZbLu+QF+AcuXK1Z82bVqmz7l7927OPPPMzAedR8THx1OsWDG/wwg5q2f48aOu0dHRnH/++Tl6zhMnTuTJXsKbN29m3759/yhr2bLlGlVtkHzbvN5J4g+gUsDnil7ZP6jqa8BrAA0aNNCUnlEIVkxMTIrPOIQbq2d4iZR6gj913bhxI8WLF8/Rcx44cCDHz5kdkmb3DUZevwe1CqgmIlVFpCBwM/CRzzEZYyJQXm6NyikZ/Y7ydIJS1QTgTmABsBGYoaob/I3KGBNpChcuTFxcnCWpNKgqcXFxFC5cOOh98noTH6o6D5jndxzGmMhVsWJFtm/fzs6dO3PsnEeOHMnQL/vcoHDhwlSsWDHo7fN8gjLGGL8VKFAgx8fbjImJybnu3j7J0018xhhjwpclKGOMMbmSJShjjDG5Up5+UDczRGQn8FsWDlEG8HeayZxh9QwvkVJPiJy6hlM9K6tq2eSFEZegskpEVqf0xHO4sXqGl0ipJ0ROXSOhntbEZ4wxJleyBGWMMSZXsgSVca/5HUAOsXqGl0ipJ0ROXcO+nnYPyhhjTK5kV1DGGGNyJUtQxhhjciVLUCkQkXYi8pOIbBaRB1NYX0hEpnvrV4pIFR/CzBZB1LWniOwUkXXecpsfcWaFiEwSkb9F5PtU1ouIPO99B+tFpF5Ox5gdgqhnCxHZF/CzfCynY8wOIlJJRJaIyA8iskFE7k5hm3D5mQZT17D4uaYopWl2I3kBooBfgH8BBYFvgerJthkAjPfe3wxM9zvuENa1J/Ci37FmsZ7NgHrA96msvxqYDwjQGFjpd8whqmcLYK7fcWZDPcsD9bz3xYGfU/h3Gy4/02DqGhY/15QWu4I6XUNgs6puUdVjwDTg+mTbXA9M9t6/B7QWEcnBGLNLMHXN81T1c2B3GptcD0xRZwVQUkTK50x02SeIeoYFVY1V1bXe+wO4ueAqJNssXH6mwdQ1bFmCOl0FYFvA5+2c/g/i5DbqJk3cB5TOkeiyVzB1BbjBayZ5T0Qq5UxoOSrY7yEcNBGRb0VkvojU8DuYrPKa1+sCK5OtCrufaRp1hTD7uSaxBGXSMweooqq1gE85deVo8p61uDHPagMvAB/4G07WiEgx4H1gkKru9zueUEqnrmH1cw1kCep0fwCBVwkVvbIUtxGR/EA0EJcj0WWvdOuqqnGqetT7OBGon0Ox5aRgfuZ5nqruV9V47/08oICIlPE5rEwRkQK4X9jvqOqsFDYJm59penUNp59rcpagTrcKqCYiVUWkIK4TxEfJtvkI6OG97wx8pt7dyjwm3boma7dvj2sDDzcfAd29nl+NgX2qGut3UNlNRM5OulcqIg1x///z3B9WXh1eBzaq6phUNguLn2kwdQ2Xn2tKbMr3ZFQ1QUTuBBbgerlNUtUNIvIEsFpVP8L9g3lLRDbjbkrf7F/EmRdkXQeKSHsgAVfXnr4FnEki8i6up1MZEdkODAUKAKjqeGAertfXZuAQ0MufSLMmiHp2BvqLSAJwGLg5j/5h1RToBnwnIuu8soeBcyG8fqYEV9dw+bmexoY6MsYYkytZE58xxphcyRKUMcaYXMkSlDHGmFzJEpQxxphcyRKUMcaYTElvgOIUtr8xYODbqeltbwnKmAAioiIyOuDzEBEZlk3HflNEOmfTsVZ6I1f/Lv8cbb5Kdhw/hfM9nIl9eorIi6GIx+QabwLtgtlQRKoBDwFNVbUGMCi9fSxBGfNPR4FOue1JfG/EkpNUtZGq1gEew42mX8dbtmbkOBmQ4QRlwl9KAxSLyHki8omIrBGRZSJykbeqD/CSqu7x9v07veNbgjLmnxKA14B7kq9IfgUkIvHeawsRWSoiH4rIFhEZKSK3iMjXIvKdiJwXcJgrRGS1iPwsItd6+0eJyP+JyCpvUN5+AcddJiIfAT+kF7iIXOddWX0jIotEpJxXPkxE3hKRL3EPmJcVkU+9ZpaJIvJbUkIWkf96ca8TkVe92EYCZ3hl76S2nVfey6vb17iHTE3keQ24S1XrA0OAl73yC4ALRORLEVkhIuleeVmCMuZ0LwG3iEh0BvapDdwOXIx78v8CVW2IG7/wroDtquCmObkGGC8ihYHeuKF4LgUuBfqISFVv+3rA3ap6QRAxfAE0VtW6uKlT7g9YVx24QlW74EaY+MxrZnkPb1QCEbkYuAnXBFMHOAHcoqoPAoe9K7RbUttO3LBYj+MS02XeOU0EETeo7b+Bmd7IF6/i5rQCN3JRNdxoJ12ACSJSMq3j2VBHxiSjqvtFZAowEDd0TDBWJY31JiK/AAu98u+AlgHbzVDVRGCTiGwBLgLaALUCrs6icf+RjwFfq+qvQcZQEZjuJYqCQOB+H6lqUl0uAzp6df1ERPZ45a1xgwGv8oZ2OwNIqRkmte0aATGqutP7Hqbj/mo2kSMfsNf7wyW57biJI48Dv4rIz7h/56vSOpgx5nTP4a5sigaUJeD9nxGRfLgkkORowPvEgM+J/PMPweRjiylu1te7Au4jVVXVpAR3MAMxv4Cb/fgSoB9QOGBdMMcRYHJAHBeq6rAsbGcijDcVyK8i8h9wg92KSG1v9Qe4qye8JuULgC1pHc8SlDEpUNXdwAxckkqylVPTjbTHG4g1g/4jIvm8+1L/An7CDdbbX9y0CojIBSJSNK2DpCKaU1NK9Ehjuy+BG71ztQFKeeWLgc4icpa37kwRqeytO54UXxrbrQSai0hpb9v/ZKIOJg8RN0DxcuBCEdkuIr2BW4DeIvItsIFTs3QvAOJE5AdgCXCfqqY56ro18RmTutHAnQGfJwAfev/xPiFjVzdJfge+BkoAt6vqERGZiLs3tVZcm9lOoEMmjj0M1/a/B/gMqJrKdo8D74pIN9wvlx3AAVXdJSKPAgu9K8TjwB3Ab7gb3+tFZK13H+q07VR1hbgu+cuBvcC6TNTB5CHePc2UnNYBwhthfbC3BMVGMzcmwohIIeCEN91KE+CVVO4ZGOMru4IyJvKcC8zwrn6O4Z5PMSbXsSsoY4wxuZJ1kjDGGJMrWYIyxhiTK1mCMsYYkytZgjLGGJMrWYIyxhiTK/0/zQm06y/OcOYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "perf_h2o.plot_uplift(metric=\"gain\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAA+yElEQVR4nO3deXwUVbbA8d8hBEIWdgw7Sdgc9iWCG0wc11EBF1AZVHAZdRQBGWdGnw6gM08FcZkRfIqKyyiig4qoKC4YFBcEEUFEZIeogOyEPXDeH7cSOk0n6ZB0utN9vp9Pfbq76lbVud2aw626da+oKsYYY0ykqRLuAIwxxphALEEZY4yJSJagjDHGRCRLUMYYYyKSJShjjDERyRKUMcaYiGQJysQkERkjIi+GO45giMgQEZnr8zlXRDK89zVE5C0R2Ski/w1flBXDvovYYgnKVCgR+YOILPD+sPwiIu+KyOnhjqs4IvKciPyzDPtni8j1fuuyRCTneI6nqsmqutr72B9IBeqp6oBgYhVnmIh8JyJ7RCRHRP4rIh2PJ55wKut3YSKbJShTYURkJPAocB/uD0lz4HGgXxjDquxaAD+qal4p9vkXMBwYBtQF2gDTgQtKe3IRqVrafULoeL4LE8lU1RZbQr4AtYBcYEAxZZ4D/unzOQvI8fm8FvgLsBjYAzyDS3TvAruBD4E6gfb12f8s7/0Y4EWfbf8FNgI7gU+A9t76G4BDwEEv/re89Y2B14BfgTXAsGLqlQ1c77fOv26KSxirgS3Ag0AVb9sQYK5f2VbAPV5ch7zYbgwUq995WwOHgR7BxlvE+W8BVnh1/z9gvN8x3gRGlva7CuK/o3L7LmyJ/CWS/vVjotspQALwRhmPcylwNlAV+AboClwHLANm4v7I33Mcx30XuBb3B20s8BLQRVUnicipuGRyN4CIVAHewv0RHgg0BT4UkeWqOqsMdbsYyASSccl2OfB0UYVVdbSIKNBKVa/0YjvFN9YAzvS2f1WGOAEuAnoC+4CTgJdE5C+qqiJSBzgH+FMIv6tCjvO7MBHOLvGZilIP2KJlv/zymKpuUtWfgE+Bear6jaruxyW/rsdzUFWdrKq7VfUArnXVWURqFVH8JKCBqt6rqgfV3QN5CrjieM7tY6yqblPV9bhLoQPLeLxA6gG/lMNx7vdi3Yf7HRTo5W3rD3yhqj8Tuu/KxABrQZmKshWoLyJVy5ikNvm83xfgc3JpDygiccD/AgOABsARb1N93CU/fy2AxiKyw2ddHO4PdSB5QLzfunjcJShfG3zer8NdGitvW4FG5XCcgli9VtNUXEL9BPgDkN9DMujvSkSaA9/7HLfUv6WJLtaCMhXlC+AA7tJQUfYAiT6fG5bhfIWO5SWhBkWU/QOuo8ZZuHtlafm7ea/+Q/5vANaoam2fJUVVzy/i+Ot9jpkvHZeEfDXzed8c+LmI4xWnpOkJPgKaikhmMWWC+R38z/My0F9EWuAu/b3mrQ/6u1LV9ep65SWXU3KyqRoqOUtQpkKo6k5gFDBRRC4SkUQRiReR34vIOK/YIuB8EakrIg2BEWU45Y9AgohcICLxwN1A9SLKpuCS51bcH+b7/LZvAjJ8Pn8F7BaRv3nP3sSJSAcROamI478CXCMiPbwu3m2A24CpfuX+IiJ1RKQZrpfdK0HWtbhYC1HVFbieky97Xd2riUiCiFwhInd4xRYBl3i/USvcPb5iqeo3uM4dTwOzVHWHt6m031V5Kva7MJHPEpSpMKr6EDASlyx+xf3reiiuizPAf4Bvcb3t3uf4/kDnn2sncDPuD+ZPuFZBUc8dvYBrzfyEu8T0pd/2Z4B2IrJDRKar6mHgQqALrlda/h/mgPesvM4AdwDP4i4ZzgSeByb5FX0T+BqXIN7xzltahWItoswwYAIwEdgBrMJ10HjL2/4IrrPIJi/Ol4I89xRcK3RK/orSflflLJjvwkQwUbVWsDHh5vVAa62qK8MdizGRwlpQxhhjIpIlKGOMMRHJLvEZY4yJSNaCMsYYE5EsQRljjIlIlqCMoWBKjO0iUt1vXZHTZIhImoio/4je/tM8iEgjEXnGm15kt4j8ICL3iEhSKWNME5GPRWSvd4yziilbXUQmi8guEdnojSSfv+1kEflARLaJyK/eVBuNfLbfJiKrvX1/FpFHfOsoIqeKyFdeXRZLhE+XYiovS1Am5olIGm4cOQX6lvOx6+JG0agBnKKqKbjBbmsDLUt5uJdxA+TWA+4CpolIUaNjjMGNXN4COAP4q4ic522rg3sGK83bvhv3jFa+GUA3Va0JdAA6456dyq/PW7jR1msD44C3vAFijSlXlqCMgatxD+c+Bwwu52OPxCWAK1V1LYCqblDV4aq6ONiDeKNPdANGq+o+VX0NWIIb3T2QwcA/VHW7qi7DDdA6xDv/u6r6X1Xdpap7cQ/tnpa/o6qu8hkJQnBjE7byPp8KbPT2P6yqL+Ieur4k2LoYEyxLUMa4BPWSt5wrIqnleOyzgNdV9UhRBbzLZDuKWB73irUHVqvqbp9dv/XW+x+vDm5A2G9LKuvpDSz1O8YfRGQXbuSHzsCTvpv9T4lraRlTrmw0cxPTvPsnLYBXVXWLiKzCDR77SDmdosTpLVS1UxDHSebYkdV3Ak2KKJu/3bdsin9BEemEGyOx0KzGqjoFmCIirXEJPH/U+C9wo5MPBKbhvquWFB5c1phyYS0oE+sGA++r6hbv8xSOXuYraZqMPJ91RZUpr+ktcoGafutq4i4fBiqbv73Ist5AsO8Cw1U14FQh3uCyS3EDzKKqW3HJbCQuaZ2Hm1yxqHEOjTlulqBMzBKRGsBlwG+9nm4bcaOMdxaRzpQ8TcYvuERUXJkPgYvFzSxbVBxLRSS3iOUJr9hSIENEfFtBnfG7NAegqtu92DoXVVbctBgf4u5T/aeo2DxV8enQoapzVPUkVa0LXAWciBu13JjyVVFzy9tiS6QtuAn2tuHmXmros3wCPAScC2wGeuDus7TBTS1/k88xXgZex13Ki/eOuQNI9bbXxY3O/h+ghbeuCfAw0KmU8X4JjAcScKOP78DNVhuo7APAHFyPvRNxCes8n/OvAm4vYt/rgRO89+1wie1hn+1dvbrWxM38+1m4f0tbonMJewC22BKuBXgPeCjA+suAjbiWw7XeH+hdwErctBlVfMrW4eiUHtuBz4DT/I7XGJjsHXM38AMwGkgsZbxpQDZu5uDlwFk+2wYBS30+V/fOuQt3KW6kz7bRuC71ub6Lz/ZnvX32eMn1QSDBZ/vLuHtaO3FTopwQ7t/SluhcbCw+Y4wxEcnuQRljjIlIlqCMMcZEJEtQxhhjIpIlKGOMMREpakaSqF+/vqalpZXpGHv27CEpqVQDTFdqsVTfWKorWH2jXbTV9+uvv96iqscMfBw1CSotLY0FCxaU6RjZ2dlkZWWVT0CVQCzVN5bqClbfaBdt9RWRdYHW2yU+Y4wxEckSlDHGmIgU0gQlIueJyHIRWSkidwTY3ltEFopInoj091nfRUS+8MYoWywil4cyTmOMMZEnZPegRCQOmIibPTQHmC8iM1T1e59i63GTqN3ut/te4GpVXSEijYGvRWSWHp1EzRhjIsahQ4fIyclh//79FXK+WrVqsWzZsgo5V3lKSEigadOmxMf7TwAQWCg7SfQAVqrqagARmYobpr8gQak3w6iIFJrMTVV/9Hn/s4hsBhrgBsc0xpiIkpOTQ0pKCmlpaYj4z+dY/nbv3k1KyjHTe0U0VWXr1q3k5OSQnp4e1D6hTFBNgA0+n3OAnqU9iIj0AKrhRl/233YDcANAamoq2dnZxxVovtzc3DIfozKJpfrGUl3B6lvRatWqRb169cjNzS25cDk4fPgwu3cHmgosslWrVo0dO3YE/VtFdDdzEWmEm6ZgsAaYMltVJwGTADIzM7Ws3S6jretmSWKpvrFUV7D6VrRly5ZRs6b/fJKhUxlbUPkSEhLo2rVrUGVD2UniJ6CZz+em3rqgiEhN4B3gLlX9spxjK+a8cOONFXU2Y4wxRQllgpoPtBaRdBGpBlwBzAhmR6/8G8ALqjothDEGNGlSRZ/RGGMq1nPPPcfQoUMBeOKJJ3jhhRcA+OGHH+jSpQtdu3Zl1apVTJkyJWwxhixBqWoeMBSYhZuF9FVVXSoi94pIXwAROUlEcoABwJMikj8l9WVAb2CIiCzyli6hitUYY2LZTTfdxNVXXw3A9OnT6d+/P9988w0bNmwIa4IK6T0oVZ0JzPRbN8rn/XzcpT///V4EXgxlbMYYE03Wrl3LhRdeyHfffQfA+PHjCzqPdO7cmTlz5pCXl8fkyZPp0aNHoX3HjBlDcnIy7dq149FHHyUuLo6PPvqIffv2sWzZMrp06cLgwYO57bbbKrROEd1JoiLt2AHjx7cJdxjGmEpuxAhYtKh8j9mlCzz66PHvv3fvXhYtWsQnn3zCtddeW5DE/J1//vncdNNNJCcnc/vtt5Odnc348eN5++23j//kZWBDHXkeeQTeeadxuMMwxphyN3DgQAB69+7Nrl272LFjR3gDCpK1oDyJieGOwBgTDcrS0imLqlWrcuTI0adxfEe18H94uCIeJi4P1oLy1KkT7giMMeb4paamsnnzZrZu3cqBAwcKXZZ75ZVXAJg7dy61atWiVq1aQR0zJSUlrA8EWwvK49+CUnXPRBljTGUQHx/PqFGj6NGjB02aNOHEE08s2Jb/cOyhQ4eYPHly0Mfs1KkTcXFxdO7cmSFDhlgniXA5dKjw57w8CHI8Q2OMiQjDhg1j2LBhhdZlZWVx5ZVX8qjftcchQ4YwZMgQwPXiy+f7Pj4+ntmzZ4co2pLZJT7PwYPFfzbGGFOxrAXlCZSgkpLCE4sxxpSXyjxosLWgPP4Jyv+SnzHGmIplCcrjn5DsEp8xxoSXJSiPtaCMMSayWILyWCcJY4yJLJagPJagjDHmqLS0NLZs2RLWGCxBefwv6dklPmNMZaWqhYY9qqwsQXn8W0wHDoQnDmOMOR5r166lbdu2XH311XTo0IHrrruOzMxM2rdvz+jRowvKpaWlMXr0aLp160bHjh354YcfANi6dSvnnHMO7du35/rrr0dVC/Z5+OGH6dChAx06dCh44Hft2rWceOKJDBkyhDZt2jBo0CA+/PBDTjvtNFq3bs1XX31V5jrZc1Aeu8RnjCkvWVlZx6y77LLLuPnmm9m7dy/nn3/+MdvzR3bYsmUL/fv3L7Qt2GeZVqxYwfPPP8/JJ5/Mtm3bqFu3LocPH+bMM89k8eLFdOrUCYD69euzcOFCHn/8ccaPH8/TTz/NPffcw+mnn86oUaN45513eOaZZwD4+uuvefbZZ5k3bx6qSs+ePfntb39LnTp1WLlyJf/973+ZPHkyJ510ElOmTGHu3LnMmDGD++67j+nTp5fqe/NnLShP166FP1sLyhhT2bRo0YKTTz4ZgFdffZVu3brRtWtXli5dyvfff19Q7pJLLgGge/furF27FoBPPvmEK6+8EoALLriAOt4I2nPnzuXiiy8mKSmJ5ORkLrnkEj799FMA0tPT6dixI1WqVKF9+/aceeaZiAgdO3YsOG5ZWAvKc/PN0K5dNikpWWRmgs9I9cYYUyrFtXgSExOL3V6/fv3jHv0hyRv+Zs2aNYwfP5758+dTp04dhgwZUmj6jerVqwMQFxdHXl7ecZ3L9zgAVapUKfhcpUqVMh234JhlPkKUyf++81tQInDLLeGLxxhjSmvXrl0kJSVRq1YtNm3axLvvvlviPr1792bKlCkAvPvuu2zfvh2AXr16MX36dPbu3cuePXt444036NWrV0jjz2ctKD8JCe51/37IzXXvH38cJk4MX0zGGFManTt3pmvXrpx44ok0a9aM0047rcR9Ro8ezcCBA2nfvj2nnnoqzZs3B6Bbt24MGTKEHj16AHD99dfTtWvXcrmEVxJLUH58W1ALF4Y3FmOMCVZaWhrfffddwefnnnsuYDnfxJKZmVlwObFevXq8//77AfcZOXIkI0eODPp8/tuOl13i8+OboKZNc+9t4kJjjKl4IU1QInKeiCwXkZUickeA7b1FZKGI5IlIf79tg0VkhbcMDmWcvvIv8Q0dCo895t7bdPDGGFPxQpagRCQOmAj8HmgHDBSRdn7F1gNDgCl++9YFRgM9gR7AaBGpkDTh0ymlwLZt1u3cGFM83wdbTWCl/Y5C2YLqAaxU1dWqehCYCvTzLaCqa1V1MeA/Jse5wAequk1VtwMfAOeFMNYC1aoV/ty4sXvdtKkizm6MqYwSEhLYunWrJaliqCpbt24lIf8yVRBC2UmiCbDB53MOrkV0vPs28S8kIjcANwCkpqaWeebI3Nxc5szJJj6+N4cOudydnLwbSOHttxfSrt2uMh0/0uTm5lbq2TZLI5bqClbfiiYiJCUlsWHDhpILlwNVRSrhzfHDhw+zZ88e1q1bF1T5St2LT1UnAZMAMjMzNdDwIqWRnZ1NVlYWdesebTG1aZPCjz/CCSd0o4yHjzj59Y0FsVRXsPpGu1ipbygv8f0ENPP53NRbF+p9y+zuu4++b9vWva5ZU1FnN8YYA6FNUPOB1iKSLiLVgCuAGUHuOws4R0TqeJ0jzvHWVYihQ+GDD+Cyy2D0aKhb1xKUMcZUtJAlKFXNA4biEssy4FVVXSoi94pIXwAROUlEcoABwJMistTbdxvwD1ySmw/c662rMGedBa+8AikpkJ5uCcoYYypaSO9BqepMYKbfulE+7+fjLt8F2ncyMDmU8QUrPR0WLw53FMYYE1tsJIkgZGTA2rUQBRNUGmNMpWEJKgjp6W4Cw59/DnckxhgTOyxBBSEjw73afShjjKk4lqCCkJ7uXi1BGWNMxbEEFYTmzd2I5qtXhzsSY4yJHZagglC9OjRtai0oY4ypSJaggpSebi0oY4ypSJaggmQP6xpjTMWyBBWkjAzXzXz//nBHYowxscESVJDS00EVghwl3hhjTBlZggpSgwbudVuFjghojDGxyxJUkPKngrdLfMYYUzEsQQUpvwX166/hjcMYY2KFJaggpaa6140bwxuHMcbECktQQapXD+Lijk4Fb4wxJrQsQQWpShVo3Bg2bAh3JMYYExssQZVC69awYkW4ozDGmNhgCaoU2raF5cvd81DGGGNCyxJUKbRpA9u3w9at4Y7EGGOinyWoUmjTxr0uXx7eOIwxJhZYgiqFli3d69q1YQ3DGGNigiWoUmjWzL2uXx/eOIwxJhZYgiqFxET3PJQNGGuMMaFXYoISkerBrCti3/NEZLmIrBSROwIdR0Re8bbPE5E0b328iDwvIktEZJmI3BnM+SqCzQtljDEVI5gW1BdBritEROKAicDvgXbAQBFp51fsOmC7qrYCHgHGeusHANVVtSPQHbgxP3mFW0aGzaxrjDEVocgEJSINRaQ7UENEuopIN2/JAhKDOHYPYKWqrlbVg8BUoJ9fmX7A8977acCZIiKAAkkiUhWoARwEdpWiXiHzm9/AypXw2WfhjsQYY6Jb1WK2nQsMAZoCDwHird8F/E8Qx24C+A4MlAP0LKqMquaJyE6gHi5Z9QN+wSXD21T1mJmYROQG4AaA1NRUsrOzgwiraLm5uSUeo1GjFKA7p58OM2d+So0ah8t0znAKpr7RIpbqClbfaBcz9VXVgAsw1nu9rKgyxS1Af+Bpn89XARP8ynwHNPX5vAqoD5wGvATEAycAy4GM4s7XvXt3LauPP/44qHJNm6q68STKfMqwCra+0SCW6qpq9Y120VZfYIEG+Lte3D2o873Lbcd0bgjST0Azn89NvXUBy3iX82oBW4E/AO+p6iFV3Qx8BmQeZxzl7o03jr63+aGMMSY0iktQ7wHbgU4isstn2S0iwdwPmg+0FpF0EakGXAHM8CszAxjsve8PzPay6XrgdwAikgScDPwQdK1CLDMTPvjAvf/22/DGYowx0arIBKWqf1HV2sA7qlrTZ0lR1ZolHVhV84ChwCxgGfCqqi4VkXtFpK9X7BmgnoisBEZytLU2EUgWkaW4RPesqi4+3kqGQmYmiMDcueGOxBhjolNxnSQAUFX/nndBU9WZwEy/daN83u/HdSn33y830PpIUrs2dOoEn38e7kiMMSY6FdfNfK73utv30l4pLvFFvVNOgXnz4MiRcEdijDHRp7hLfKd7rym+l/aCvcQXC04+GXbtgh8i5u6YMcZEjyIv8YlI3eJ21ADPJcWak092r337uod3jTHGlJ/i7kF9jRvRQQJsUyAjJBFVIq1bu9dVq2D2bPjd78IbjzHGRJMiE5SqpldkIJVRlSrw5ZeuJfXEE5CV5dYZY4wpO/tzWkY9e8KIEfDf/8LkyeGOxhhjooclqHLw8MNulPNp08IdiTHGRA9LUOVABM4/H2bNgvfeC3c0xhgTHYJKUCJyuohc471vICJ2f8rPHXdA06YwZIjrem6MMaZsgplRdzTwNyB/Vtt44MVQBlUZNWkCL78MmzbB449DLIyEb4wxoRRMC+pioC+wB0BVfwZSQhlUZXXaaW5K+DvvhDPOgP794Sf/8duNMcYEJZgEddAbYVyhYHRxE4AIfPjh0c+vvQZ//Wv44jHGmMosmAT1qog8CdQWkT8CHwJPhTasyisjA95//+jnN96ALVvCF48xxlRWJSYoVR2Pm4L9NaAtMEpVHwt1YJXZ2We7+XaXLoV9+9w9KWOMMaUTTCeJkcD33vxQt6vqBxUQV1Ro1w4uuAAmTHCJasMGOHw43FEZY0zlEMwlvhTgfRH5VESGikhqqIOKJn/5i5sW/r77oHlz+Mc/wh2RMcZUDsFc4rtHVdsDtwCNgDki8mEJuxlP795u9t1//tN9fvLJ8MZjjDGVRWlGktgMbAS2AieEJpzoIwJ33XX088aNsHdv+OIxxpjKIph7UDeLSDbwEVAP+KOqdgp1YNGkXz83Tt/f/uY+z5kT3niMMaYyCKYF1QwYoartVXWMqn4f6qCijQhceimMHg3JyW7cvnfeCXdUxhgT2YpMUCKSP637g8B6Eanru1RMeNGlRg14/XX3/pZb7FKfMcYUp7gW1BTv9Wtggff6tc9ncxzOPtuN07duHTzwQLijMcaYyFVkglLVC73XdFXN8F7zl6CmexeR80RkuYisFJE7AmyvLiKveNvniUiaz7ZOIvKFiCwVkSUiknAc9YtIv/0tDBoEY8fCypXhjsYYYyJTMJ0kPgpmXYAyccBE4PdAO2CgiLTzK3YdsF1VWwGPAGO9faviRky/yevingUcKumclcmDD0L16jBsmBt1whhjTGHF3YNK8O411ReROj73n9KAJkEcuwewUlVXq+pBYCrQz69MP+B57/004EwREeAcYLGqfgugqltVNarGYGjUCO69F959F958M9zRGGNM5KlazLYbgRFAY9x9J/HW7wImBHHsJsAGn885QM+iyqhqnojsxHVlbwOoiMwCGgBTVXWc/wlE5AbgBoDU1FSyyzgJU25ubpmPURodOwoZGd256aaqJCR8RULCkQo7N1R8fcMpluoKVt9oFzP1VdViF+DWksoUsV9/4Gmfz1cBE/zKfAc09fm8CqgP3A6s8d4nAl8AZxZ3vu7du2tZffzxx2U+Rml98okqqN59d4WfOiz1DZdYqquq1TfaRVt9gQUa4O96MEMdPSYiHUTkMhG5On8JIvf9hHuGKl9Tb13AMt59p1q4kSpygE9UdYuq7gVmAt2COGel06sXXHUVjBsHK1aEOxpjjIkcwU75/pi3nAGMw82wW5L5QGsRSReRasAVwAy/MjOAwd77/sBsL5vOAjqKSKKXuH4LRO0DwuPGQUIC3HqrdZgwxph8wYwk0R84E9ioqtcAnXEtnWKpah4wFJdslgGvqupSEblXRPIT3DNAPRFZCYwE7vD23Q48jEtyi4CFqhq1Yy80bOg6TMyaBdOnhzsaY4yJDMV1ksi3T1WPiEieN7rEZgpfuiuSqs7EXZ7zXTfK5/1+YEAR+76I62oeE265BZ55BkaMgHPOgaSkcEdkjDHhFUwLaoGI1MZN8/41sBDXacGUo6pVYeJEWL/ezR1ljDGxrsQWlKre7L19QkTeA2qq6uLQhhWbevWCq692CWr/frj/fqhWLdxRGWNMeBT3oG43/wWoC1T13psQGOc97fXww3DGGXCkYh+NMsaYiFFcC+qhYrYp8LtyjsUAqamwejX84Q/w+efw73+7+1LGGBNrikxQqnpGRQZijkpPd8mpb1+48074/e+hbdtwR2WMMRWrxHtQRT2Uq6ovlH84Jp8ITJoE7dvDkCEwdy7ExYU7KmOMqTjB9OI7yWfpBYwhuAd1TRk1agQTJsCXX8JDxV1wNcaYKBRML75bfT97Xc6nhiogU9jAgfDaa/D3v8NZZ0E3655ijIkRwbSg/O0B0ss7EBOYCDzxhOs8cdFFsGlTuCMyxpiKEcxYfG+JyAxveRtYDrwR+tBMvgYN4I03ICfHDYt0yinwxRewe3e4IzPGmNAJZqij8T7v84B1qpoTonhMEbp3h5dfhjFjYOFCOPVUSEyEadNcLz9jjIk2wUy3MUdV5wDf4AZ93evNtGsq2OWXw7JlbjikqVMhI8N1RX/wQVi3LtzRGWNM+QrmEt8NIrIRWAwswI3HtyDUgZmipaa6ZDVnjmtJ/fWv0LKlS1TGGBMtgukk8Regg6qmqWqGqqarakaoAzMlq1sXsrPhq6/g3HNdojrpJHjzTZtXyhhT+QWToFYBe0MdiDk+Ii4pzZgBjz4KO3e63n49ergWljHGVFbBJKg7gc9F5EkR+Xf+EurATOnExcHw4fD99zB5suuOnpUFl14Kq1aFOzpjjCm9YBLUk8Bs4Evc/af8xUSgqlXhmmvghx/gH/9ws/S2awd/+YtrXRljTGURTIKKV9WRqvqsqj6fv4Q8MlMmiYlw993w449uZPSHHoLWrd1Dv3l54Y7OGGNKFkyCetfryddIROrmLyGPzJSLxo3h2WdhwQL4zW/gT3+CLl3g/ffDHZkxxhQvmAQ1EO8+FEcv71k380qmWzfX4++112DfPtfr7847O/LDD+GOzBhjAgvmQd30AIt1M6+EROCSS1xHinHjYMmSWnTsCMOGwdat4Y7OGGMKs/mgYlD16q7TRKtW83j//dOYOBFefBFGj4abb4b4+HBHaIwxNh9UTKtT5xD/93+waBFkZrqp5Tt0gLfftgd9jTHhF8wlvlt9lj8C3YDkYA4uIueJyHIRWSkidwTYXl1EXvG2zxORNL/tzUUkV0RuD7I+5jh07Oi6o7/9trsM2KcPnHMOLFkS7siMMbEsZPNBiUgcMBH4PdAOGCgi7fyKXQdsV9VWwCPAWL/tDwPvHkeMppRE4IILXFL617/g669db78bb4TNm8MdnTEmFoVyPqgewEpVXa2qB3Gz8PbzK9MPyH+mahpwpoiId96LgDXA0qBqYspFfLzrNLFyJdx6qxuVolUr16niwIFwR2eMiSWiJdxsEJHf+nwMej4oEekPnKeq13ufrwJ6qupQnzLfeWVyvM+rgJ7AfuAD4GzgdiBXVcf7nQIRuQG4ASA1NbX71Kllm4k+NzeX5OSgrl5GhWDqu359DZ54oiVffFGfRo32ceONq+nd+1fcPyMqD/tto5vVt3I744wzvlbVzGM2qGrABWgFnBZg/WlAy6L28ynXH3ja5/NVwAS/Mt8BTX0+rwLq4yZJvMxbNwa4vaTzde/eXcvq448/LvMxKpPS1PeDD1Q7dFAF1V69VBcsCF1coWC/bXSz+lZuwAIN8He9uEt8jwK7Aqzf5W0ryU9AM5/PTb11AcuISFWgFrAV14oaJyJrgRHA/4jIUEzYnHUWfPONGyrphx/cCOpDhsDPP4c7MmNMtCouQaWq6jH9uLx1aUEcez7QWkTSRaQacAUww6/MDGCw974/MNtLqL3UzT+VhkuG96nqhCDOaUKoalXXaWLFCvcc1csvu/H97r0X9u8Pd3TGmGhTXIKqXcy2GiUdWFXzgKHALNxU8a+q6lIRuVdE8p+jegaoJyIrgZHAMV3RTeSpVQvGjnXTz//+9+4B34EDITc33JEZY6JJcSNJLBCRP6rqU74rReR6gpxuQ1VnAjP91o3yeb8fGFDCMcYEcy5T8TIyYNo0GD/ezebbvj1cdx306wedOlHpOlIYYyJLcS2oEcA1IpItIg95yxzcs0vDKyQ6UyncfrubvbdFCxgzxj0/lZ7uuqt/9BEcOhTuCI0xlVGRCUpVN6nqqcA9wFpvuUdVT1HVjRUTnqksevWCTz5xnSaeesq1oJ56ynWuaNDAzUn1yis2aaIxJnglDharqh8DH1dALCYKNGwI11/vlj174IMPYMYMeOst16kiPt5NRd+3r1uaNw93xMaYSHU8Qx0ZE5SkJLjoIjcaxcaNMHeuG5B2/Xo3SkWLFm6eqjFjXBd2G6DWGOPLEpSpEHFxcNppbsikH35wy7hxbmr6e+91iapFCxg61M32e/BguCM2xoSbJSgTFm3bumep5s51ravJk12SmjzZzfbboAFccQVMmQI7doQ7WmNMOFiCMmF3wglwzTUwfbqb2XfGDLjsMjdF/aBBLlmdeSb8+9+wdm2YgzXGVBhLUCai1Kjh5qN66inXI/Dzz1039l9+geHDXff1zp1h1ChYsMDuWxkTzSxBmYhVpQqccgrcfz98/z38+KN7KLh2bfjf/3XjATZrBn/6E7z3nk0HYky0sQRlKo3WreHPf3YPBW/aBM89Bz17wn/+44Zcql8fBgyAF1+EbdvCHa0xpqxKfA7KmEhUvz4MHuyW/fth9mx48013/2raNNdrsFcv96xVP/9pMo0xlYK1oEyll5AA558PTz4JP/0E8+bB3/4GW7bAyJHQsiVce20md90FX30FR46EO2JjTDAsQZmoUqUK9Ojh7lEtWeKmrn/kEahV6xBjx7pLgk2bumlDZs60aUKMiWSWoExUa9nSjV7xyCPfsnmzu191+unu+aoLLnCXCi+9FJ5/3rW4jDGRw+5BmZhRty5ceaVbDhyAjz9296xmzIDXX3etr9NOc/es+vWDVq3CHbExsc1aUCYmVa8O550Hjz8OGza4Z6ruusuNtn777a7HYLt2cOed8MUXdt/KmHCwBGVingh07+7GBPz2W1izBv71L2jc2D13deqp7v0f/+hGZd+3L9wRGxMbLEEZ4yctzU22+OGHsHmzu1+VlQWvvuq6rder50Zpf/ZZt90YExp2D8qYYtSpAwMHuuXgQfeQcP7zVm++6Vpfp57q7ln17esGwTXGlA9rQRkTpGrV4OyzYcIEWLcOFi6E0aNh717461/hxBPd8te/wmefweHD4Y7YmMrNEpQxx0EEunZ1CWrhQpewJkxwMwQ/+qjryt6oEVx7rWtp7d0b7oiNqXwsQRlTDpo3h1tucZMt/vorTJ3qWluvv+7uV9Wr5y4BPv20G0fQGFOykCYoETlPRJaLyEoRuSPA9uoi8oq3fZ6IpHnrzxaRr0Vkiff6u1DGaUx5qlULLr8cXnrJJauPPoIbboDFi11PwEaN3H2rBx6AZctsyhBjihKyThIiEgdMBM4GcoD5IjJDVb/3KXYdsF1VW4nIFcBY4HJgC9BHVX8WkQ7ALKBJqGI1JlTi4+F3v3PLo4+64ZfyO1nceadbmjSBDh3ca8OGhZfUVPeakuIuKxoTS0LZi68HsFJVVwOIyFSgH+CboPoBY7z304AJIiKq+o1PmaVADRGprqo244+ptESgUye3/P3vkJPjnqv6/HM339WSJe7yX6DOFTVqHJu8/JNY/vuEhIqvmzGhEMoE1QTY4PM5B+hZVBlVzRORnUA9XAsq36XAwkDJSURuAG4ASE1NJTs7u0wB5+bmlvkYlUks1TdS6/qb37gl35EjsGtXPNu2VWPbtmps316t4H3+8s037nXXrviAx0xOPkTt2t2pX387desepG7dg9Spc6jgff5Sq9ZB4uIqqKIhFqm/b6jESn0j+jkoEWmPu+x3TqDtqjoJmASQmZmpWVlZZTpfdnY2ZT1GZRJL9Y3Guh486O5xbdzov8SzZMl2Dh8+gfXrYf582L372P2rVIEGDY5thQVaateO7EuM0fj7FidW6hvKBPUT0Mznc1NvXaAyOSJSFagFbAUQkabAG8DVqroqhHEaUylVq+buWzUJcHc2O/t7srJOKPi8Z4+7fHhsMju6/ocf3OvBg4HPVVISy9+elBTCSpuYEsoENR9oLSLpuER0BfAHvzIzgMHAF0B/YLaqqojUBt4B7lDVz0IYozExISkJMjLcUhxV2LEjcALLX9avdxM/bt4cuAdicnJw98tOOMElPmOKErIE5d1TGorrgRcHTFbVpSJyL7BAVWcAzwD/EZGVwDZcEgMYCrQCRonIKG/dOapqI58ZE0IibninOnUK3xsLJC/PzaFVVMts40b47jvXzX779sDHqFev6ATmu9Sr5y5JmtgS0ntQqjoTmOm3bpTP+/3AgAD7/RP4ZyhjM8aUTdWqRxNI587Fl92/37W4ikpkmza5aU1++SXwaPFxcS55FZXAfvmlFo0bW5f8aBPRnSSMMdEhIcGNttG8efHlVCE3t+j7ZPnL4sVuXV5e/p5dGTHi6LmCuVeWmuq675vIZQnKGBMxRFwLKCXFTRpZnCNHYNs2l7BmzVpEw4ZdjklsK1fC3LnuUmQgtWqVfK+sYUPX27Gq/bWscPaVG2MqpSpVoH59t2zZsoPiel0fOlRUl/yjyzffuNddu47dX+Rol/yS7pfVqWOXGMuLJShjTNSLj3ezIjduXHLZvXsDd/zwXbd8uXs9EGBsm/j44J4tS011PR5N0SxBGWOMj8RESE93S3FUYefO4u+V5eTAggWug8iRI8ceIykp+Ptlsdgl3xKUMcYcBxE3wkbt2m6iyuIcPlxyl/zvv4ePP3b31QKpW/do4hL5DW+9FbilVr9+9HTJtwRljDEh5ttNvlOn4sseOFB0l/z8BLdmTU2++CLwRJhxce4h6JLulTVsCDVrRvb9MktQxhgTQapXh2bN3FKU7Ox5ZGVlBdUlf8kS93q0S/5R+V3yg7lfFo4u+ZagjDGmkkpOhlat3FKcI0fcaB6BElj+snq1m/plyxbfIawOA7uAXJKSdlO3bi61a+8mPb0b9evXITl5Axde2Iyzzw5N/SxBGWNMFDhy5Ah79+5l9+7dJCcnk5KSwq5du/j000/Jzc0lNzeX3bt3k5ubS9++ffnDHzqxdOlS/v73v5OfgFJTc0lKyuWBB56iTZuzmD79He69tx/gBhzeswc2bIAdOz7m0KEsNm36ga1bLUEZY0zUUVXWrVtXkDjyXzMyMujSpQt79uzh/vvvL1ifX6Z79+5kZWWxfv16evbsSW5uLnv27EG9ps9jjz3G0KFDWbduHRdeeOEx523SpAmdOnXi4MGDrFixoiChNWrUiJSUFFq1qkPXrlC7dkfq1n2U5OTkgiUlJYVOnTpRuzbs2vXbgF3ty4slKGOMKUFeXl5BgsjNzSU+Pp6WLVsC8Prrr7Nly5ZCSaZ9+/Zcc801AJx//vmFtufm5jJo0CAmTJjAkSNHSA/Qn33kyJF06dIFVeX+++8vSAz5rwe8rFCzZk369OlTaFtycjKnn346AC1btuTLL78stD0pKYlqXp/1rl27smTJkiLrnZ6ezvDhw4vcXrNmaPu+W4IyxkQVVWX//v3U8O7qr1ixgo0bNxZqgdSoUYOBAwcCMH78eL777rtCLZg2bdrw7LPPAu6P+KJFiwqd45xzzmHWrFkA3Hbbbaxfv75gW2JiIgMGDChIUFWqVKFevXq0aNGiIFH06tULgLi4OJ5//nkSExMLJZGGDRsCkJSURF5eHuLX1S5/Nt3atWszadKkIr+LxMREevb0n8i88rAEZYwJq0OHDrFr165CCSI3N5ezzjoLgNmzZ7NgwYJC248cOcIzzzwDwB133MGUKVMKtXIaN25MTk4OACNGjGDmzEKTKtC6deuCBDV37ly++eabggSRnJxM7dq1C8pec8017Nixo1ACaebTxW7OnDlUq1aN5ORkkpKSiIuLK3Sut99+u9j6X3311UVu809MscYSlDEmaIcPH2bPnj3k5uZSv359qlWrxvr161m8ePEx91GGDRtG7dq1eeONN3jhhReO2b548WJq1arF3Xffzbhx444514EDB6hWrRqvv/46EydOREQKEoRvAmnUqBGtWrUiIyOjIIk0aNCgYPuYMWO47bbbCl0Cq1mzZsH26dOnF1vnYcOGFbs9LS0tqO/OlJ4lKGOi3IEDB/jpp5+OSRCnnHIKTZo0Yfny5Tz//POFtu3evZtx48bRqVMn3njjDW666SZyc3PZ6/Nk6MKFC+natSszZ87kT3/60zHnveKKK6hduzbbt29n9erVBS2Tpk2bkpKSUnBDv0+fPjRp0qTQTfjk5OSClsgDDzzA2LFjSUxMDNiiGD58OJ07dyariNFiTzrppHL4Fk04WIIyJkIcOnSo0H2SunXr0rBhQ3Jzc5k+ffoxCeSiiy6id+/erFq1iuuvv77Q5bEdO3bw5JNPMmjQIObPn19wz8PXtGnTuPTSS1m7di3jxo0jJSWl0GWu/fv3A9CsWTMuvvjiY27EN2nSBICLLrqI7t27F9o3OTmZqt78FNdeey3XXnttkfU+/fTTC27qB5JsI6rGLEtQxhyHI0eOcOjQIapXrw7AokWLjrmP0rJlS7Kysjhy5Ai33nrrMS2Yyy+/nBEjRrBjxw4aNmxY0DMr3z333MOoUaPYuXMnV111VaFtSUlJtGrVit69exMfH8/hw4dJTU2lZcuWJCcns3PnTlp5T2+2bduW559//pgWSosWLQB3w//QoUNF3u/IzMwkMzOzyO+iYcOGBTf1jSlPlqBMTDh48CC//vproQQRHx9fcPnnpZdeIicnp1ASadOmDX/7298AOPfcc1m9enXB9j179jBgwABeffVVAM444wx27NhR6JyDBw8mKyuLKlWq8Prrr1O9evVCrZDExETAtRCGDx9eqAWTkpJCx44dAUhNTeXHH38s2JaYmEgVn9FAmzdvzieffFLo3NnZ2QW9txo0aGA34k2lZAnKRJz83lh79+6lsTeBz5IlS1izZk2hS2BxcXGM8Ob5Hjt2LJ9//nmhBNSoUSM++ugjAP785z/z3XffFTpPZmYm8+fPB+Chhx7im2++oUqVKgWJ4ODBgwVl09PTqVevXqEWSCefUT+nTp1KfHx8oe2+N/J/+eWXIutbtWpVxo4dW+z21iVNL2tMFLIEZcrF7t272bRp0zFPvF900UVUq1aNDz/8kA8//LDQw465ubm88847xMXFcddddzFp0iRyc3ML7n0kJCSwb98+AMaNG8eLL75Y6Jz16tUrSFAbNmxg/fr1JCcnFzxz4tu76uKLL+aGG24o1ELx7ek1e/ZsqlevTkJCQsAWxRNPPFFs/c8999zj+dqMMcWwBBVjDh48WJBE1qxZQ2ZmJsnJyaxbt445c+YcM2bXn//8Z5o2bcrbb7/N+PHjj7mPsmjRIjIyMnj88ce54447jjnfL7/8QsOGDfn00095+OGHj7nRfuDAARITE+nQoQMDBgwotC05ORlVRUQYNWoUI0aMKLQ9KSmp4DwTJkwott6/+93viuzlBRRq7RhjIkNIE5SInAf8C4gDnlbVB/y2VwdeALoDW4HLVXWtt+1O4DrccLrDVHVWKGONRIcPH+bIkSPEx8ezb98+li5dekwC6d27N23btmXVqlWMHz++0Pbdu3fzwAMPcMYZZzBr1iz69OnDoUOHCp2jWbNmnHXWWcybN4/BgwcXrM9/5mTQoEE0bdq0oEtw/lhdvsOmAFxwwQUF23y316tXD4DRo0dzzz33FFnXgQMHFjw4GYhd4jIm9oQsQYlIHDAROBvIAeaLyAxV/d6n2HXAdlVtJSJXAGOBy0WkHXAF0B5oDHwoIm1U9XCo4i0rVWXfvn3HPA1/wgkn0Lp1aw4cOFBwCcu3FdK3b18uvvhiNm7cyLnnnlvo8tfevXt59NFHGT58OGvWrAn4PMekSZNo27YtO3fu5LXXXivUwqhTp05BV9+MjAxuv/32Qj251q1bV3Aj/rzzzmPFihWFbsT7Xurq06cPffr0KbL+HTp0oEOHDkVurxItU3waYypMKFtQPYCVqroaQESmAv0A3wTVDxjjvZ8GTBD3V7EfMFVVDwBrRGSld7wvQhgvS5YsYefOnYUSSEZGBpdeeikA1113HZs3by7USunbt2/BU/D5l6R8DRs2jH/9618cOXKk4In0uLi4gpZG/o32hIQE0tLSjnmWJL8nVvPmzXnzzTePuQRWv359ALp168bmzZuLrFvr1q257777Cq3Lzs4mNTUVcINO+j5db4wx4RbKBNUE2ODzOQfwH7WwoIyq5onITqCet/5Lv32bhC5U54EHHuDnn38utK5Pnz4FCWrZsmXs27evYCiV9PT0ghGNRYQHH3yQhISEQq2UjIwMwCWgX3/9laSkpIA34mvXrs2bb75ZZGzJycn07du3PKtrjDERTfz/xV9uBxbpD5ynqtd7n68CeqrqUJ8y33llcrzPq3BJbAzwpaq+6K1/BnhXVaf5neMG4AaA1NTU7lOnTi1TzIsWLSIxMZEaNWoUvCYkJETt5anc3NyYeUo/luoKVt9oF231PeOMM75W1WOeBg9lC+onoJnP56beukBlckSkKlAL11kimH1R1UnAJIDMzEwtrpdWsMrjGJVFdnZ2zNQ3luoKVt9oFyv1DWXTYD7QWkTSRaQartPDDL8yM4D8rmP9gdnqmnQzgCtEpLqIpAOtga9CGKsxxpgIE7IWlHdPaSgwC9fNfLKqLhWRe4EFqjoDeAb4j9cJYhsuieGVexXXoSIPuCWSe/AZY4wpfyF9DkpVZwIz/daN8nm/HxhQxL7/C/xvKOMzxhgTuaLz7r8xxphKzxKUMcaYiGQJyhhjTESyBGWMMSYihexB3YomIr8C68p4mPrAlnIIp7KIpfrGUl3B6hvtoq2+LVS1gf/KqElQ5UFEFgR6mjlaxVJ9Y6muYPWNdrFSX7vEZ4wxJiJZgjLGGBORLEEVNincAVSwWKpvLNUVrL7RLibqa/egjDHGRCRrQRljjIlIlqCMMcZEpJhMUCJynogsF5GVInJHgO3VReQVb/s8EUkLQ5jlIoi6DhGRX0VkkbdcH444y4OITBaRzd5EmIG2i4j82/suFotIt4qOsTwFUd8sEdnp89uOClSushCRZiLysYh8LyJLRWR4gDJR8RsHWdeo+n0DUtWYWnBTf6wCMoBqwLdAO78yNwNPeO+vAF4Jd9whrOsQYEK4Yy2n+vYGugHfFbH9fOBdQICTgXnhjjnE9c0C3g53nOVY30ZAN+99CvBjgP+eo+I3DrKuUfX7BlpisQXVA1ipqqtV9SAwFejnV6Yf8Lz3fhpwpohIBcZYXoKpa9RQ1U9w84oVpR/wgjpfArVFpFHFRFf+gqhvVFHVX1R1ofd+N7AMaOJXLCp+4yDrGvViMUE1ATb4fM7h2B++oIyq5gE7gXoVEl35CqauAJd6l0OmiUizigktLIL9PqLJKSLyrYi8KyLtwx1MefEuu3cF5vltirrfuJi6QpT+vvliMUGZwt4C0lS1E/ABR1uOpvJbiBvjrDPwGDA9vOGUDxFJBl4DRqjqrnDHE0ol1DUqf19fsZigfgJ8WwlNvXUBy4hIVaAWsLVCoitfJdZVVbeq6gHv49NA9wqKLRyC+e2jhqruUtVc7/1MIF5E6oc5rDIRkXjcH+yXVPX1AEWi5jcuqa7R+Pv6i8UENR9oLSLpIlIN1wlihl+ZGcBg731/YLZ6dyUrmRLr6nd9vi/uWne0mgFc7fX0OhnYqaq/hDuoUBGRhvn3TkWkB+7/98r4Dy3A9dADngGWqerDRRSLit84mLpG2+8bSNVwB1DRVDVPRIYCs3C93Car6lIRuRdYoKozcP9h/EdEVuJuQl8RvoiPX5B1HSYifYE8XF2HhC3gMhKRl3E9m+qLSA4wGogHUNUngJm4Xl4rgb3ANeGJtHwEUd/+wJ9EJA/YB1xRSf+hle804CpgiYgs8tb9D9Acou43Dqau0fb7HsOGOjLGGBORYvESnzHGmErAEpQxxpiIZAnKGGNMRLIEZYwxJiJZgjLGGHNcShqwOED5y3wGwJ1SUnlLUCamiYiKyEM+n28XkTHldOznRKR/OR1rnjdi9XopPPp8WnkcP8D5/uc49hkiIhNCEY+JWM8B5wVTUERaA3cCp6lqe2BESftYgjKx7gBwSaQ9ge+NYFJAVXuqahdgFG50/S7esrY0xymFUicoE3sCDVgsIi1F5D0R+VpEPhWRE71NfwQmqup2b9/NJR3fEpSJdXnAJOA2/w3+LSARyfVes0Rkjoi8KSKrReQBERkkIl+JyBIRaelzmLNEZIGI/CgiF3r7x4nIgyIy3xuk90af434qIjOA70sKXET6eC2rb0TkQxFJ9daPEZH/iMhnuAfOG4jIB95lladFZF1+QhaRK724F4nIk15sDwA1vHUvFVXOW3+NV7evcA+XGjMJuFVVuwO3A49769sAbUTkMxH5UkRKbHlZgjIGJgKDRKRWKfbpDNwE/Ab3xH8bVe2BG8/wVp9yabhpTy4AnhCRBOA63BA8JwEnAX8UkXSvfDdguKq2CSKGucDJqtoVN5XKX322tQPOUtWBuBEmZnuXVabhjUYgIr8BLsddcukCHAYGqeodwD6vhTaoqHLihsm6B5eYTvfOaWKYuMFtTwX+642A8SRubitwIxe1xo1+MhB4SkRqF3e8mBvqyBh/qrpLRF4AhuGGjAnG/Pwx3kRkFfC+t34JcIZPuVdV9QiwQkRWAycC5wCdfFpntXD/4x4EvlLVNUHG0BR4xUsU1QDf/Waoan5dTgcu9ur6nohs99afiRsceL43pFsNINBll6LK9QSyVfVX73t4BfevZBO7qgA7vH/I+MvBTSB5CFgjIj/i/rufX9zBjDHwKK5lk+SzLg/v/xERqYJLAvkO+Lw/4vP5CIX/4ec/lpjiZnu91ec+Urqq5ie4PaWI+THcbMgdgRuBBJ9twRxHgOd94mirqmPKUM7EOG9KkDUiMgDcoLci0tnbPB3XesK7xNwGWF3c8SxBGQOo6jbgVVySyreWo9OP9MUbiLWUBohIFe++VAawHDd475/ETaeAiLQRkaTiDlKEWhydSmJwMeU+Ay7zznUOUMdb/xHQX0RO8LbVFZEW3rZD+fEVU24e8FsRqeeVHXAcdTCVmLgBi78A2opIjohcBwwCrhORb4GlHJ3FexawVUS+Bz4G/qKqxY6+bpf4jDnqIWCoz+engDe9/9Heo3Stm3zrga+AmsBNqrpfRJ7G3ZtaKO6a2a/ARcdx7DG4a/3bgdlAehHl7gFeFpGrcH9MNgK7VXWLiNwNvO+1EA8BtwDrcDe6F4vIQu8+1DHlVPVLcV3yvwB2AIuOow6mEvPucQZyTAcIb6T1kd4SFBvN3JgoJyLVgcPe9CunAP9XxD0CYyKKtaCMiX7NgVe91s9B3PMoxkQ8a0EZY4yJSNZJwhhjTESyBGWMMSYiWYIyxhgTkSxBGWOMiUiWoIwxxkSk/wd3DI/B/AdAhgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "perf_h2o.plot_uplift(metric=\"lift\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Qini value and Average Excess Cumulative Uplift (AECU)\n",
    "\n",
    "Qini value is calculated as the difference between the Qini AUUC and area under the random uplift curve (random AUUC). The random AUUC is computed as diagonal from zero to overall gain uplift. \n",
    "\n",
    "The Qini value can be generalized for all AUUC metric types. So AECU for Qini metric is the same as Qini value, but the AECU can be also calculated for Gain and Lift metric type. These values are stored in ``aecu_table``.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "\n",
       "#h2o-table-7.h2o-container {\n",
       "  overflow-x: auto;\n",
       "}\n",
       "#h2o-table-7 .h2o-table {\n",
       "  /* width: 100%; */\n",
       "  margin-top: 1em;\n",
       "  margin-bottom: 1em;\n",
       "}\n",
       "#h2o-table-7 .h2o-table caption {\n",
       "  white-space: nowrap;\n",
       "  caption-side: top;\n",
       "  text-align: left;\n",
       "  /* margin-left: 1em; */\n",
       "  margin: 0;\n",
       "  font-size: larger;\n",
       "}\n",
       "#h2o-table-7 .h2o-table thead {\n",
       "  white-space: nowrap; \n",
       "  position: sticky;\n",
       "  top: 0;\n",
       "  box-shadow: 0 -1px inset;\n",
       "}\n",
       "#h2o-table-7 .h2o-table tbody {\n",
       "  overflow: auto;\n",
       "}\n",
       "#h2o-table-7 .h2o-table th,\n",
       "#h2o-table-7 .h2o-table td {\n",
       "  text-align: right;\n",
       "  /* border: 1px solid; */\n",
       "}\n",
       "#h2o-table-7 .h2o-table tr:nth-child(even) {\n",
       "  /* background: #F5F5F5 */\n",
       "}\n",
       "\n",
       "</style>      \n",
       "<div id=\"h2o-table-7\" class=\"h2o-container\">\n",
       "  <table class=\"h2o-table\">\n",
       "    <caption>AECU values table: All types of AECU value</caption>\n",
       "    <thead><tr><th>uplift_type</th>\n",
       "<th>qini</th>\n",
       "<th>lift</th>\n",
       "<th>gain</th></tr></thead>\n",
       "    <tbody><tr><td>AECU value</td>\n",
       "<td>5699.5299236</td>\n",
       "<td>0.0171021</td>\n",
       "<td>6651.3994210</td></tr></tbody>\n",
       "  </table>\n",
       "</div>\n"
      ],
      "text/plain": [
       "AECU values table: All types of AECU value\n",
       "uplift_type    qini     lift       gain\n",
       "-------------  -------  ---------  ------\n",
       "AECU value     5699.53  0.0171021  6651.4"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "perf_h2o.aecu_table()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Normalized AUUC\n",
    "\n",
    "To get normalized AUUC, you have to call ``auuc_normalized`` method. The normalized AUUC is calculated from uplift values which are normalized by uplift value from maximal treated number of observations. So if you have for example uplift values [10, 20, 30] the normalized uplift is [1/3, 2/3, 1]. If the maximal value is negative, the normalization factor is the absolute value from this number. The normalized AUUC can be again negative and positive and can be outside of (0, 1) interval. The normalized AUUC for ``auuc_metric=\"lift\"`` is not defined, so the normalized AUUC = AUUC for this case. Also the ``plot_uplift`` with ``metric=\"lift\"`` is the same for ``normalize=False`` and ``normalize=True``.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAABHjklEQVR4nO3deZxN9f/A8dfbPmXsvlKSPSFrKW1Ii7Qq8lWELL+StFiSZCn6CpWKaC8tJJUk2jDRQihLSJYU2ce+xcy8f398znAbs9wxc+fcuff9fDzuY+4959xz3597ue/7Oedz3h9RVYwxxphwk8fvAIwxxpjUWIIyxhgTlixBGWOMCUuWoIwxxoQlS1DGGGPCkiUoY4wxYckSlIk4IjJYRN71O45giEhHEfku4PEBEank3Y8Rkc9EZK+IfOhflOFPRC4XkdV+x2GylyUok21E5A4RWeR9yW4RkZkicpnfcaVHRN4SkaFZeH6ciHRJsayJiGw6lf2pamFVXe89bAWUAUqqautgYhWnp4j8KiIHRWSTiHwoIuefSjy5harOU9Vz/Y7DZC9LUCZbiMjDwGjgKdyXanngJeBmH8PK7c4BflfVhEw853ngAaAnUAKoBkwFrs/si4tIvsw+x5hspap2s1uWbkBR4ADQOp1t3gKGBjxuAmwKeLwB6AMsAw4Cr+MS3UxgP/ANUDy15wY8/yrv/mDg3YB1HwJbgb3AXKCmt7wbcAw46sX/mbf8TOAjYAfwB9AznXbFAV1SLEvZNsUljPXATmAkkMdb1xH4LsW2VYAhXlzHvNj+L7VYU7xuVSARaBhsvGm8/n3AGq/t44BRKfbxKfBwZt+rIP4d5QWe8d6jP4AeXjz5vPWdgFXev4f1wP9l8O+pt/fvaS/wAVDI7/8rdsvczXpQJjs0AgoBn2RxP7cBV+N+9d+IS079gdK43n7PU9zvTNyX93+An4H3AFT1Fe/+CHWH1m4UkTzAZ8BS4CygGfCgiFx7qo3ytAQuAOrjepV3p7exqg7C9UY/8GJ7OWWsqTytGe5L+qcsxnoLcBFQA5gItBERARCR4sA1wKQQvFddgeuAurj36ZYU67cDNwBFcMnqORGpn87+bgeaAxWB2rhkbHIRS1AmO5QEdmrmDkWl5kVV3aaqfwPzgAWq+ouqHsElv3qnslNVfUNV96vqP7jeVR0RKZrG5hcCpVX1CVU9qu580KvAf0/ltQM8raq7VPUv3KHQtlncX2pKAluyYT//82I9jPscFLjcW9cK+FFVN5P979XtwPOquklVdwPDA1eq6uequk6db4GvAuJKzQuqullVd+ESad1TjMv4xI4xm+wQD5QSkXxZTFLbAu4fTuVx4czuUETyAsOA1rieWJK3qhTu0E9K5wBnisiegGV5cV/UqUkA8qdYlh93OC7QxoD7f+IOjWW3eKBsNuzneKyqqiIyCZdQ5wJ3AMkjJIN+r0SkPLAyYL+pfZZn8u/3KfA+InIdMAjXw84DnAYsT6cdWwPuHyI077kJIetBmezwI/APJx+SCXQQ94WS7IwsvN6/9uUlodJpbHsH7pDaVbhzZRWSn+b9TVnOfyPwh6oWC7jFqmqLNPb/V8A+k1XEJaFAZwfcLw9sTmN/6clo6oFZQDkRuSCdbYL5HFK+zkSglYicgzv095G3POj3SlX/8g5NFk4jOYHr/ZULeHz8PRORgt7rjgLKqGoxYAYnPkcTgSxBmSxT1b3AQGCsiNwiIqeJSH4RuU5ERnibLQFaiEgJETkDeDALL/k7UEhErheR/MAAoGAa28bikmc87ov5qRTrtwGVAh7/BOwXkUe865DyikgtEbkwjf1/AHQSkYbeEO9qwEPApBTb9RGR4iJyNm6U3QdBtjW9WP9FVdfgRk5O9Ia6FxCRQiLyXxHp5222BLjV+4yqAJ0zelFV/QU3cOE14EtV3eOtyux7lZHJwAMicpaIFAMeCVhXAPcZ7wASvN7UNaf4OiaXsARlsoWqPgM8jEsWO3C/rnvghjgDvIM7mb4Bd+7gVL6gk19rL9Ad94X5N65XkNZ1RxNwvZm/cYeY5qdY/zpQQ0T2iMhUVU3EnYivixtJlvzFnOo5K1X9EugHvIk7ZDgDeBt4JcWmnwKLcQnic+91M+tfsaaxTU9gDDAW2AOsww3Q+Mxb/xxuJOA2L873gnzt93G90PeTF2T2vQrCq7h/G8uAX3DvZQKQqKr7vbZNBnbjesbTTvF1TC4hqjZhoTGhJCIKVFXVtX7Hkpt4vaTxqnqO37EYf1gPyhgTFrzDhC1EJJ+InIUbEJHVSxdMLmYJyhgTLgR3gfJu3CG+VbhzmyZK2SE+Y4wxYcl6UMYYY8KSJSgT9QIrkovInSLyVTbvv4KIqBVfNSZzLEGZkBORDSKyXUROD1jWRUTifAwrVar6nqr6cn2Nlyh3exelplye5pQeaSXAlNNziEhZEXndmwplv4j8JiJDAj+XIOOsICJzROSQt4+r0tm2hIh8ICLxIrJTRN4TkSIB6y8RkZ+8eJZJwPQsItJURJZ7w+rjReQTb/BEUPs2uZ8lKJNT8uIuUM0S72LYiPt3KyIVcHXlFLgpBPsvgav4EQM0UtVYXGHeYkDlTO5uIm4QQ0ngMWCKiKRVyWMoUBxXXaMyrkL94ICYPsNVdy8GjAA+E1eQFtx1a9d6VSPOxFVYHxfMvk1kiLj/6CZsjQR6exUCTuL9kl4obvbYhSJyScC6OBEZJiLf42qqVfJ6DN1FZI336/tJEaksIj+IyD4RmSwiBbznFxeR6SKyw+uhTBeRcmnEcXyGWxHpK27yxeTbMRF5y1tXNKA38reIDBVXcgmvosIo71f9eoKbi+ku3EXEbwEdgnpHM+dh3DQV7VR1A4CqblTVB1R1WbA7EVcpoz4wSFUPq+pHuHp4t6XxlIrAVFXd511g/QlQ01t3CbBVVT9U1URVfRd3kfetXnzbvKK0yRJxU5EEs28TASxBmZyyCDcXUe+UK7xf0p8DL+B+lT8LfC4iJQM2a4+bvymWE3XurgUaABcDfXHVG9rharjV4kTF8Dy4Sg/n4OrgHcZVW0iXqiZPbVEYOA/35ZlcAeMtXJWDKrgq69cAyYfhuuIqLNTDTbHRKqPXwiWo97zbtSJSJojnZMZVwMeqmpTWBt4htj1p3F7yNqsJrPcqOyRbStqJYSxwg/cjoTgukc0MfNmUYeA+u+SYyosrRnsY929nRMC2Ge3b5HKWoExOGgjcn8rhoOuBNar6jqomqOpE4DfcnFDJ3lLVFd765ErhI7xfzyuAX4GvVHW992t6Jt70HKoar6ofqeoh74t1GNA42KBFJAZXsul5VZ3pJY8WwIOqelBVt+NKCCVPM3E7MNrroewC/pfB/i/DJc/JqroYV57ojmDjC1KGU3Goau0UhV8Db929zQpzchX4vbgfDqn5GVdHL967JeLqBYI75HimiLQVV7uxA+5Q3fFitl6R2WK46vMDcP8ugtm3iQCWoEyOUdVfgem42nWBzuTk6t9/4ibBS7aRkwU1PYe4wqgvi8ifIrIPN21EseRDckF4HVitqk97j8/BTamxJbmHAbyMmxAxuT0pp9dITwdcct3pPX6ffx/my2hKj4SAZWltk11TcRzATRgYqAju8GFqJuOK+8Z6263Dm65DVeNxleYfxn12zXEzJ59UV9FL9G8Dn8qJwSBp7ttEBktQJqcNwh0CC0w+m3Ff+oHK4wq8JsvKFeW9gHOBi1S1CHCFtzzDqRrEVQGvxr+rfm/EVUgvFdDDKKKqyYe5tnDy9Bpp7T8G1+NqLCJbRWQrrhp6HRGp422W0ZQeW3CJKL1tvgFapjfARERWpDjnFngb7222AncOMLDHVMdbnpq6wMteT/MAMB7X+wRAVb9V1QtVtQTuMG51XJX01OTD/QhITpDp7tvkfpagTI7yCqZ+wL+nb58BVBORO8TVYWuDm258eja9bCyuR7XHO981KJgniStW2hNo6c0um9yGLbiq28+ISBERyeMN0Eg+bDgZ6Cki5bxzIyl7jIFuwR2aqoH7wq2LO981D3deCjKY0sOrKv4RMExESnqHy9p6+0w+J/Ms7ov9bXHzOiFuWotnRaS2t5+agXM2pbjd423zO64i+yBxU3m0xE2nnjxHVEoLgS7i6uzF4M4jHh+UISL1vHiL4OZ62uhViEdEbhWRc733t7TXhl+83lSG+za5nyUo44cngOPX3niHem7A9XTicQMebgg45JVVo3HDq3fiRsp9EeTz2uAmQlyVSk/iLtz5j5W42nFTOHEI7VXgS9zggZ+Bj9N5jQ7Am965lq3JN9wgjjvFzVIczJQe3YFduC/o7bipTq5X1W1w/BDZJbie1gIR2Y+b4HAvkNkq6//FDf5Inpa9larugOMXOgf2pu7G9ew24XrElfj34cu+uM9lI+79axmw7izcZ7UfN1IwKcX6jPZtcjmrxWeMMSYsWQ/KGGNMWLIEZYwxJixZgjLGGBOWLEEZY4wJS7mu/H+pUqW0QoUKWdrHwYMHOf30TBVwztWirb1gbY4W1ubIsHjx4p2qelLB4VyXoCpUqMCiRYuytI+4uDiaNGmSPQHlAtHWXrA2Rwtrc2QQkVSrrdghPmOMMWHJEpQxxpiwZAnKGGNMWLIEZYwxJixZgjLGGBOWLEEZY4wJSyFLUCLyhohsF5Ff01gvIvKCiKz1ppquH6pYjDHGZK+kpNC/Riivg3oLN2XAhDTWXwdU9W4XAeO8v8YYY0Ls6FHYvx/27XN/A28ZLYuLc/v49FO46abQxRiyBKWqc0WkQjqb3AxMUDffx3wRKSYiZb3J4IwxxgRQhUOHID6+AL//nrmEktqyo0eDe91ChSA29sStSBGoWBH++GMPHTvuYtWqSpQpE5o2+1lJ4izcJGXJNnnLTkpQItINN1smZcqUIS45fZ+iAwcOZHkfuUm0tReszdEi3NucmCgcOpT3+O3w4XwcOpSXgwdP3HfL83LwYL7j9w8dyhfwvHwcPuyWJyUJbt7JtIkoMTGJnHZaIjExiZx+egIxMYnExiZSpkyCtyyR005LOL6duyUcvx8Tc+J+vnz/njPw6NGjTJ06lbfffo+9e+vx7LPjue66rSF5/3JFqSNVfQVv9tALLrhAs1rmIxJLhaQn2toL1uZokd1tVoUjRzLfG0lr2eHDwb1u/vz/7qHExkLp0icvi42FLVt+p0GDav9aFrjdaacJefLkIxRf75MmTaJfv378+eefXHPNNfTq9TTXXFMdqJ7trwX+Jqi/gbMDHpfzlhljTNCSkuDAgexJKPv3Q0JCcK972mknJ49y5U5ellqSSbmsYMHg2xsXt5kmTaqd2puVRT/99BMlSpTgtdde46qrrgr56/mZoKYBPURkEm5wxF47/2RMdEg+QZ/VhLJnz2VB91Ly5Ek9UZQtm/mEUrgw5M0b2vcoHCxevJhHHnmERx99lGbNmjFs2DAKFixInjw5c4VSyBKUiEwEmgClRGQTMAjID6Cq44EZQAtgLXAI6BSqWIwxWZN8gj47eij79gV/gr5gwZMTxX/+A1WquPt79myhRo2zg+q1xMSASGjfp0ixfv16BgwYwMSJEylZsiTx8fEAxMTE5GgcoRzF1zaD9QrcF6rXNybaJSZmX0I5cCD4615SJonYWDfqK5geSsrl+fOn/1pxceto0uTs9DcymTJkyBCGDRtGvnz56N+/P3379qVo0aK+xJIrBkkYEw1U4Z9/Tj2hbNnSANUTy4I99JUv38nJolgxKF8+c4e9YmPh9NPdoTSTuxw8eJCCBQuSL18+SpcuTYcOHRg8eDBnnXWWr3FZgjImhxw4AHPnuosc16yBbdtOTjKZPUEfeCtV6h8qVIjNVEJJPkFvh76iU0JCAm+++SaDBg3iySefpHPnznTv3t3vsI6zBGVMiBw7Bj/9BN98427z57sEVKCAO4dStiyccUbmeijJJ+jzpfI/Ny7u16gbZm5Ojaoybdo0Hn30UVatWkWjRo2oUaOG32GdxBKUMdlEFX791SWjWbPg229dr0kEGjSAXr3gqqvg0kvdCXtj/NK1a1def/11qlWrxscff8wtt9yChGE32hKUMVnw558uGX3zDcye7Q7bAVStCu3bQ7Nm0LQplCjhb5zGrF69mrJly1KkSBFuv/12LrjgAjp37kz+jEai+MgSlDGZEB8Pc+ac6CWtXeuWlynjekfNmrlb+fL+xmlMsi1btjBkyBBee+01HnvsMYYMGcI111zjd1hBsQRlTDoOHYLvvjvRS/rlF3coLzYWGjeGHj1cQqpZ0wYamPCyf/9+Ro4cyTPPPMPRo0e59957ue++3HVljyUoYwIkJMCiRScS0g8/uItK8+eHRo1gyBCXkC68MONrdIzxU7du3Zg0aRK33347w4YNo0qVKn6HlGmWoExUU4XffjtxyG7OHDfsG6BuXbj/fnfo7vLL3TU+xoQrVeXDDz+kYcOGVKhQgYEDB/Lwww9z4YUX+h3aKbMEZaLO33+f6CHNmgWbN7vllSpBmzYuITVt6qpJG5MbzJkzh759+7Jo0SIeeeQRhg8fznnnned3WFlmCcpEvD173MWxyQnpt9/c8lKlTgxqaNbMJShjcpNly5bRr18/Zs6cydlnn81bb71Fu3bt/A4r21iCMhHnyBH4+edifP21S0qLFrk6cqed5gY2dOnieknnn29leUzuNm7cOH788UdGjBjB/fffT6FChfwOKVtZgjK5XmKiG12XfNjuu+/gyJG65M0LF10EAwa4hHTRRa6KgzG51e7duxk/fjwFCxakUaNGDB06lGHDhlEiQi+0swRlch1Vd/1RcgmhOXNg9263rlYtuOce+M9/lnPffedTpIi/sRqTHY4cOcKLL77IU089xd69e6lfvz6NGjWiZMmSfocWUpagTK6wdavrISX3kjZudMvPPhtuucX1kK680tW2A4iLi7fkZCLC5MmT6d27Nxs3buS6667jtttuo3Pnzn6HlSMsQZmwtH+/q2WXPLDh11/d8uLFXSLq398NbKhSxS6QNZHHTZcHIsL69espU6YMb7/9Nk2bNiUuLs7f4HKQJSgTFo4eddW+kxPSggXu3FKhQu4apOS6dnXrRsdU2yZ6LVq0iL59+9K1a1fatm1Lr1696Nu3b45Nsx5OLEEZXyQlwbJlJw7ZzZ3rygrlyeOqNDzyiDts16iRS1LGRLp169bx2GOP8cEHH1C6dGmSvCmMw7mYa6hZgjI55o8/TgxsmD0bdu50y6tXh7vvdgmpcWM3m6sx0WT48OEMHDiQ/Pnz8/jjj9O7d2+K2ElUS1AmdHbscIkouZf0xx9u+ZlnQosWJy6Q9XlWaWN8cfDgQfLkyUNMTAyVK1fm7rvvZtCgQZQtW9bv0MKGJSiTbQ4ehHnzTvSSli51y4sWdaWDHn7Y9ZLOPdcGNpjolZCQwOuvv87gwYPp2bMnjz76KK1bt6Z169Z+hxZ2LEGZTDl40J0niolxU5ovXHhiYMOPP7plBQq4WWOHDXM9pAYNUp+i3JhooqpMnTqVRx99lNWrV3PppZfStGlTv8MKa/a1YYK2YoVLPGedBRdfDB9+6IaDi0D9+q6H1KyZ2+a00/yO1pjw8sADD/Diiy9SvXp1pk6dyk033RSW06yHE0tQJihLlsDVV8Peve62Zg20awfXX29TmhuTllWrVlG8eHHOOOMM2rdvz/nnn0+nTp3IZ4cUghJ9A+tNpi1c6JJQTIxLTAcOuJ7TG2/AbbdZcjImpc2bN9O1a1dq1arFsGHDALjwwgvp2rWrJadMsHfKpOv77+G669zcSLNmQYUKfkdkTPjat28fI0aM4NlnnyUhIYH777+fAQMG+B1WrmUJyqRpzhy48UZ3zmnWLChXzu+IjAlvjzzyCOPHj6dt27YMHTqUSjbJWJbYIT6Tqi+/dNcqVajgauJZcjLmZElJSUycOJFly5YB0L9/fxYtWsT7779vySkbhDRBiUhzEVktImtFpF8q68uLyBwR+UVElolIi1DGY4IzbRrcdJOr8BAXd6JCuDHmhFmzZtGwYUPuuOMOXn75ZQDOPvtsGjRo4HNkkSNkCUpE8gJjgeuAGkBbEamRYrMBwGRVrQf8F3gpVPGY4EyZ4gY+1KnjqkCUKuV3RMaEl6VLl9K8eXOuuuoqduzYwYQJE3jxxRf9DisihbIH1RBYq6rrVfUoMAm4OcU2CiQXnCoKbA5hPCYD770Hbdq4mWe/+cZNbWGM+bePPvqIn376iWeeeYbVq1fTvn37qKw0nhMked6RbN+xSCuguap28R63By5S1R4B25QFvgKKA6cDV6nq4lT21Q3oBlCmTJkGkyZNylJsBw4coHDhwlnaR24STHtnzDiDUaPOpW7dPQwb9isxMYk5FF1oRNtnDNbmUNm7dy/vvfce9erVo1GjRhw+fJiEhARiY2ND+rppicTPuWnTpotV9YKTVqhqSG5AK+C1gMftgTEptnkY6OXdbwSsBPKkt98GDRpoVs2ZMyfL+8hNMmrv2LGqoHrttaqHDuVMTKEWbZ+xqrU5ux06dEiHDx+uRYsWVRHRwYMHh+y1MiMSP2dgkabyfR/KfunfwNkBj8t5ywJ1BiYDqOqPQCHAznrkoOeeg/vuc8PJP/3UXYxrTLSbMmUK1apVo1+/flx22WUsW7aMQYMG+R1W1AllgloIVBWRiiJSADcIYlqKbf4CmgGIyHm4BLUjhDGZAE895erntWrlBkcULOh3RMb4R1WPTxK4b98+ypYtS1xcHNOnT6dWrVo+RxedQpagVDUB6AF8CazCjdZbISJPiMhN3ma9gK4ishSYCHT0unsmhFRh4EB47DG4806YONFVIDcmWi1YsICmTZsyZswYADp27MiCBQto3Lixz5FFt5BWklDVGcCMFMsGBtxfCVwayhjMv6m66dRHjoTOneHllyFvXr+jMsYfa9asoX///kyZMoXSpUvTqVMnABuVFybsU4giqvDAAy45de8Or7xiyclEr9GjR1OjRg1mzpzJoEGDWLduHR06dPA7LBPAavFFiaQkuPdel5QefhhGjbJZbU30OXDgAImJiRQtWpQ6derQtWtXBg4cyBlWLiUsWQ8qCiQmCp06ueT02GOWnEz0OXbsGOPGjaNKlSoMHjwYgKZNm/LSSy9ZcgpjlqAi3LFjMGzYeUyYAE8+CUOHWnIy0UNVmTJlCjVr1qR79+5Uq1aNNm3a+B2WCZIlqAj2zz9w++0wZ85/GDkSbFoaE20ee+wxWrduTYECBfjss8/49ttvufjii/0OywTJzkFFqMOHXdHXmTOhZ8819O5d1e+QjMkRK1asICYmhkqVKtGxY0eqVKlChw4dyGsjgnId60FFoIMHXWWIL75w551atkxZwMOYyLNp0yY6d+5M7dq1efzxxwGoVq0ad999tyWnXMoSVITZv99N0T5nDrz9NnTt6ndExoTWnj17ePTRR6latSrvvvsuDzzwAC+88ILfYZlsYIf4IsiePS45LVzoqkPcfrvfERkTeiNGjGD48OHceeedDB06lAoVKvgdkskmlqAiRHw8XHMNLF/u6urdcovfERkTGsnTrJcvXx6A3r1707p1a+rVq+dzZCa7ZXiIT0RuFZE1IrJXRPaJyH4R2ZcTwZngbNsGTZvCihWuIrklJxOpvvrqKxo0aEC7du14/fXXAShRooQlpwgVzDmoEcBNqlpUVYuoaqyqFsnwWSZHbN4MTZrAunXw+efuEJ8xkWbJkiVcffXVXHvttezZs4f33nuPN954w++wTIgFc4hvm6quCnkkJtNUoV072LTJjdi7/HK/IzImNL777jt++eUXnnvuOe69914K2twwUSGYBLVIRD4ApgL/JC9U1Y9DFZQJzrvvutF648ZZcjKRZefOnQwbNow6derQsWNHunXrRvv27SlatKjfoZkcFMwhviLAIeAa4EbvdkMogzIZ27ULevWCiy+Gbt38jsaY7HHo0CH+97//UblyZV544QV+//13AAoUKGDJKQpl2INS1U45EYjJnAEDXJL6+muwqWtMJJg6dSr33Xcfmzdv5sYbb+R///sfNWvW9Dss46M0E5SI9FXVESLyInDSLLeq2jOkkZk0rVkDr74K//d/UKeO39EYc+pUlYSEBPLnz0++fPkoX748kyZN4nI7Zm1IvweVPDBiUU4EYoKzfTtccQWcdhr07+93NMacuvnz59O3b1+uuOIKhg4dyvXXX8/111+PWLl940kzQanqZ97ft3MuHJORBx5wh/a+/x7OOsvvaIzJvNWrV9O/f38+/vhjypQpQ8eOHQEsMZmTZHgOSkRKA48ANYBCyctV9coQxmVS8ddfMHky9O4NF1zgdzTGZN748ePp0aMHMTExDBkyhIcffpjChQv7HZYJU8GcXn8Pd7ivIjAE2AAsDGFMJg0jRrgBEd27+x2JMcHbv38/27dvB+CSSy7hnnvuYd26dQwcONCSk0lXMAmqpKq+DhxT1W9V9W7Aek857Pvv3dQZHTvCOef4HY0xGTt69Chjx46lSpUqPPTQQwDUrl2bMWPG8J///Mfn6ExuEEyCOub93SIi14tIPaBECGMyKaxeDbfeCuXLu16UMeFMVZk8eTI1atSgR48e1KhRgwcffNDvsEwuFEwliaEiUhToBbyIu3D3oZBGZf6lY0dX1mjqVChe3O9ojEnf8OHD6d+/P7Vq1eLzzz/nuuuuswEQ5pQEc6HudO/uXqBpaMMxKS1dCvPnw3PPQa1afkdjTOqWL1+OqlK7dm06depE2bJlad++vc1ka7IkmFF8qU1NuRdYpKqfZn9IJtC4cVCwINx1l9+RGHOyjRs38vjjjzNhwgSaN2/OjBkzOOOMM44PHTcmK4I5B1UIqAus8W61gXJAZxEZHbLIDFu2wIQJcOedUMLO+pkwsnv3bvr27UvVqlWZNGkSvXr14t133/U7LBNhgjkHVRu4VFUTAURkHDAPuAxYHsLYol7yj9B+/XwNw5iTvPHGG4waNYr27dvzxBNPcI4NLTUhEEwPqjgQeLHC6UAJL2H9k/pTHBFpLiKrRWStiKT6NSsit4vIShFZISLvBx15hFu3Dr76Ch57DKpW9TsaE+0SExOZMGEC06ZNA6B79+4sWbKEt99+25KTCZlgZ9RdIiJvishbwC/ASBE5HfgmrSeJSF5gLHAdrgpFWxGpkWKbqsCjuB5aTeDBU2lEJHr3XXdRrp17Mn5SVb744gvq169Phw4dmDBhAgAxMTHUrl3b5+hMpMswQXkX6V6Cm7DwE+AyVX1NVQ+qap90ntoQWKuq61X1KDAJuDnFNl2Bsaq623ut7afQhoizcye8/DI0aABnn+13NCZaLVmyhKuuuorrrruOAwcOMGnSJCZPnux3WCaKiOpJM2lkz45FWgHNVbWL97g9cJGq9gjYZirwO3ApkBcYrKpfpLKvbkA3gDJlyjSYNGlSlmI7cOBA2JZY2bcvH71712HDhtMZMuRXGjXaleV9hnN7Q8XanHWzZ8/mhRde4K677uLGG28kf/782bbv7GKfc2Ro2rTpYlU9ucKoqobkBrQCXgt43B4Yk2Kb6bheWX5crb+NQLH09tugQQPNqjlz5mR5H6Hw99+qtWurFiigOmNG9u03XNsbStbmzNu+fbv27NlTR44cqaqqiYmJunfv3myILHTsc44MuMuWTvq+D+VcrH8DgQeoynnLAm0CpqnqMVX9A9ebisohAStXQqNGsH49TJ8O113nd0QmWhw8eJBhw4ZRuXJlxo4dy7Zt2wDIkycPRYoU8Tk6E82CSlAicpmIdPLulxaRikE8bSFQVUQqikgB4L/AtBTbTAWaePstBVQD1gcXeuSYNw8uvRSOHoW5c+Hqq/2OyESLzz//nKpVqzJgwACaNWvGr7/+ysiRI/0OyxgguEoSg4ALgHOBN3GH497FnTdKk6omiEgP4Evc+aU3VHWFiDyB685N89ZdIyIrgUSgj6rGZ6VBuc3q1XDVVVCxInzxBVSo4HdEJtKpKkeOHCEmJoZSpUpRqVIlPvzwQy69NN3/0sbkuGAu1G0J1AN+BlDVzSISG8zOVXUGMCPFsoEB9xV42LtFpZdfdn9nz4Yzz/Q3FhP5vv/+e/r27UvNmjV55ZVXuOiii5g3b54VczVhKZhDfEe9RKIA3vVPJhscOgSffAKXX27JyYTWb7/9RsuWLbnssstYv349F1100fF1lpxMuAomQU0WkZeBYiLSFXdx7quhDSs6vPMObNgA99zjdyQmkr399tvUqlWLWbNm8eSTT7J27Vo6d+7sd1jGZCiY6TZGicjVwD7ceaiBqvp1yCOLcDt2wODBULOmm4zQmOy0b98+9uzZQ/ny5WnatCk9evTgscceo3Tp0n6HZkzQMuxBicjDwEpV7aOqvS05ZZ0q3H037NoF773nShoZkx2OHj3KCy+8QOXKlenSpQsA5cuXZ/To0ZacTK4TzFdjLPCViMwTkR4iUibUQUW6cePctU5PPw116vgdjYkESUlJzJ49m/POO48HHniA2rVr89RTT/kdljFZEkwtviHqCrneB5QFvhWRNIvEmvStWAG9esG110LPnn5HYyLF+PHjefLJJylcuDAzZ87km2++4YILTq4cY0xuEsww82Tbga1APPCf0IQT2Y4cgTvugNhYeOstO7Rnsmbp0qUcOHCASy+9lPbt2/P333/zxBNP2DTrJmIEcw6qu4jEAbOAkkBXVbU6+6egXz9YtgzefBPOOMPvaExu9eeff3LXXXdRr149+vbtC0BsbCxXX321JScTUYLpQZ0NPKiqS0IcS0T74gt4/nm4/364/nq/ozG50a5du3jqqacYM2YMAH369KGfTbdsIliaCUpEiqjqPmCk97hE4HpVzfo8EFFi2zbo0AFq1YIRI/yOxuRW06ZN49lnn6VDhw4MGTKE8uXL+x2SMSGVXg/qfeAGYDGuikTg5eYKVAphXBEjeUj53r0waxYUKuR3RCa3SExM5J133kFE6NChA+3bt6dhw4bUqFEj4ycbEwHSPAelqjd4fyuqaiXvb/LNklOQxoyBGTNg1CjXgzImI6rKjBkzqFevHp06deKDDz4AIG/evJacTFQJZpDErGCWmZMtXw59+rhzTvfd53c0JjdYunQpV155Jddffz2HDh3igw8+4PPPP/c7LGN8kd45qELAaUApESnOiUN8RYCzciC2XO/ee6FYMXjjDbB6nCYYu3btYsWKFYwZM4auXbtSoEABv0MyxjfpnYP6P+BB4Ezceajkr9h9wJjQhpX7xcfDDz/AwIHwH7tqzKRh+/btPPnkk8TExDBixAiaNm3Khg0bOO200/wOzRjfpXcO6nlVrQj0TnEOqo6qWoJKx969rvekCjfd5Hc0JhwdOHCAJ554gsqVKzNu3DiOHj2Km9UGS07GeIKpZv6iiNQCagCFApZPCGVguVmbNvDll9CqFdSr53c0Jtx89dVXdOjQga1bt3Lrrbfy1FNPce655/odljFhJ9gp35vgEtQM4DrgO8ASVBoWLoSzzoIPP/Q7EhMuVJX9+/dTpEgRKlSowHnnncfHH39Mo0aN/A7NmLAVTDW4VkAzYKuqdgLqAEVDGlUudviwm0ajWze/IzHhYt68eVxyySV06NABgGrVqjF79mxLTsZkIJgEdVhVk4AEESmCKxp7dmjDyr2WLHF/q1TxNQwTBlauXMlNN93EFVdcwV9//cUNN9xw/DyTMSZjwdTiWyQixXDTvC8GDgA/hjKo3Gz+fPfXZjqIblOmTKFNmzYULlyYp556igceeMAGPxiTScEMkuju3R0vIl8ARVR1WWjDyr2+/BLOPReqVfM7EpPT9u7dy9atWzn33HNp1qwZvXv3pk+fPpQqVcrv0IzJldI8xCci9VPegBJAPu++SWHVKvjqK7jtNr8jMTnpn3/+YfTo0VSuXJk77rgDVaV48eI8/fTTlpyMyYL0elDPpLNOgSuzOZZc7/HH4fTT4cEH/Y7E5ISkpCQmTZrEY489xoYNG7jqqqt4+umnESsbYky2SDNBqWrTnAwkt1u0CD76CAYNgtKl/Y7G5ISJEyfSrl076taty1dffcXVV1/td0jGRJRgroO6K7XldqHuvw0YACVLwsMP+x2JCaUlS5awdetWmjdvTuvWrSlQoAC33XYbefIEMyDWGJMZwfyvujDgdjkwGLACPgG+/dYNjnj0UShSxO9oTChs2LCBdu3aUa9ePfr06YOqUqBAAVq3bm3JyZgQCWYU3/2Bj70h55NCFVBuNHAgnHkmdO+e8bYmd4mPj2fYsGGMHTuWPHny0K9fPx555BE7z2RMDjiVn34HgYrBbCgizUVktYisFZF+6Wx3m4ioiOS6q4f27YO5c13liJgYv6Mx2W3+/Pk8//zztGvXjjVr1vC///2PYsWK+R2WMVEhmHNQn+FG7YFLaDWAyUE8Ly8wFrga2AQsFJFpqroyxXaxwAPAgsyFHh6SK0fYhbmRITExkbfffpu9e/fy0EMP0aJFC1avXk0VKw1iTI4Lpgc1Cjfk/Bngf8AVqppmbyhAQ2Ctqq5X1aO4w4I3p7Ldk8DTwJHgQg4vX30FefOClVXL3VSV6dOnU7t2bTp37sxnn32GqiIilpyM8YkEWxvMq8N3vMelqrsy2L4V0FxVu3iP2wMXqWqPgG3qA4+p6m0iEoebe2pRKvvqBnQDKFOmTINJk7J2CuzAgQMULlw4S/sASEwUunVrQP78SYwf/3OW9xcq2dXe3CQzbf7jjz8YPXo0y5Yto1y5cnTp0oUrrrgi151nss85OkRim5s2bbpYVU8+DqWq6d5wiWErsAFYD/wBrA/iea2A1wIetwfGBDzOA8QBFbzHccAFGe23QYMGmlVz5szJ8j5UVT//XBVU33wzW3YXMtnV3twkmDYnJSWpqurSpUu1bNmy+tJLL+nRo0dDHFno2OccHSKxzcAiTeX7PphisX2AWqq6M5NJ8W/+XfW8nLcsWSxQC4jzfqmeAUwTkZs0lV5UOPr9d/f3hhv8jcNkzrZt2xgyZAiHDh3irbfeonbt2mzYsIECBQr4HZoxJkAw56DWAYdOYd8LgaoiUlFECgD/BaYlr1TVvapaSlUrqGoFYD6Qa5ITwJw5UKaMu0DXhL/9+/czePBgKleuzKuvvkpsbCxJSUkAlpyMCUPB9KAeBX4QkQXAP8kLVbVnek9S1QQR6QF8CeQF3lDVFSLyBK47Ny2954e7HTtg+nTo2xdy2amKqPTtt99y++23s337dlq3bs2wYcOoWrWq32EZY9IRTIJ6GZgNLAeSMrNzVZ2BmyY+cNnANLZtkpl9+236dEhKgtat/Y7EpEVV2bVrFyVLlqR69eo0aNCAwYMH07BhQ79DM8YEIZgElV9VrcJcAFV49VUoXx7q1fM7GpOaJUuW8Mgjj5AvXz6+++47ypQpw4wZMzJ+ojEmbARzDmqmiHQTkbIiUiL5FvLIwtjChfDjj3Z4Lxz9+uuv3HDDDTz00ENs3ryZLl262DTrxuRSwfSg2np/Hw1YpkCl7A8nd5gwAQoVgnbt/I7EBJoxYwY33ngjsbGxdOvWjdGjRxNj9aeMybUy7EGpasVUblGbnFThs8+geXMoWtTvaMyePXv4+Wd3kXTTpk0ZMGAA69ato23btpacjMnlbD6oTPr5Z/jrLzf/k/HPkSNHGDt2LMOGDaN48eL8/vvvxMTEMGTIEL9DM8ZkE5sPKpOGDXNzPt12m9+RRKekpCTeeecdqlevTu/evWnYsCEfffQRefPm9Ts0Y0w2s/mgMmHXLvj0U+jTB0pE9TAR/8ycOZO77rqL+vXr8/rrr9OsWTO/QzLGhEgwgyRSCno+qEjzzTfu2qeborb/6I/Fixezdu1a2rRpQ4sWLfjss89o0aKFzWRrTIQL2XxQkWjyZChdGuw6z5yxfv16BgwYwMSJE6lUqRK33XYb+fLl4wYrfmhMVAimBzUq4H4C8KeqbgpRPGFr2zb45BPo1QvynUq/0wRt586dDB06lJdeeol8+fLRv39/+vbtSz57442JKmn+jxeRKkAZVf02xfJLRaSgqq4LeXRh5Icf3OG9li39jiTyrV+/njFjxtCpUycGDx7MWWed5XdIxhgfpPeTdDT/vjg32T5v3Y0hiCdsffghFC8O9ev7HUnkSUhI4M0332TDhg0MGzaMhg0bsmHDBsqVK+d3aMYYH6V3lrmMqi5PudBbViFkEYWhnTvdxbktW0LBgn5HEzlUlU8//ZTatWvTrVs35s6dy7FjxwAsORlj0k1QxdJZF1WX6D/9NBw65M4/meyxatUqLr/8cm655RaSkpL4+OOPmTt3Lvnz5/c7NGNMmEgvQS0Ska4pF4pIF2Bx6EIKL/HxMH48/Pe/UKOG39HkfomJiQCcfvrpbN68mZdffplff/2Vli1bIlZ51xgTIL1zUA8Cn4jInZxISBcABYCoGSrw6qtw4AD07+93JLnbli1bGDJkCH/++SczZ86kfPnyrFmzxipAGGPSlGaCUtVtwCUi0hSo5S3+XFVn50hkYeKHH+Dcc6FmTb8jyZ3279/PyJEjeeaZZzh69Cj33nsvx44dI3/+/JacjDHpCqbU0RxgTg7EEnb++QfmzIG2bTPe1pxswYIF3HjjjezYsYM2bdowbNgwKleu7HdYxphcwmrFpOOXX9zhvebN/Y4k91BVNm/eDECNGjVo3LgxP/30E5MmTbLkZIzJFLs0Px3z5rm/F13kbxy5xZw5c+jbty8HDhxg+fLlxMbG8uGHH/odljEml7IeVDomTnR196yQQfqWLVtGixYtuPLKK9m2bRv9+vWzEXnGmCyzHlQa9u6FJUtg8GC/Iwlv8+bNo3HjxhQtWpQRI0Zw//33U6hQIb/DMsZEAOtBpWHaNDe9e9OmfkcSfnbv3s087/jnJZdcwvDhw1m3bh19+vSx5GSMyTaWoFJx6JCb0r1WLbj0Ur+jCR9Hjhxh1KhRx6e+OHLkCHnz5qVv376UsBkcjTHZzBJUKt5/H/76C55/HmxOPFf94e2336ZatWr06dOHiy++mG+++cZ6S8aYkLKv3xS2boWuXd3EhHZ4z1mwYAEdO3bkjDPOYPbs2cycOZPatWv7HZYxJsLZIIkURo50f4cOhWgeiLZo0SIWL17M//3f/3HJJZcwZ84cGjdubKPzjDE5xnpQKcybBxdcAN26+R2JP9auXUubNm248MILefLJJzl8+DAATZo0seRkjMlRIU1QItJcRFaLyFoR6ZfK+odFZKWILBORWSJyTijjyciGDbBwIbRu7WcU/ti5cyf3338/5513HtOnT+fxxx9n5cqVxMRE1cwqxpgwErIEJSJ5gbHAdUANoK2IpJyw4hfgAlWtDUwBRoQqnmBMmeL+RmOC2r17N6+99hqdO3dm7dq1PPHEExQpUsTvsIwxUSyU56AaAmtVdT2AiEwCbgZWJm/gFaJNNh9oF8J4MjR5MjRoABUr+hlFzjh27BhvvPEGixcv5pVXXqFq1aps3LiRUqVK+R2aMcYAoU1QZwEbAx5vAtKratcZmJnaChHpBnQDKFOmDHFxcVkK7MCBAyftY+vWgixc2Ihu3dYRF7cx9SfmUoHtVVW+++47Xn31VTZu3EitWrX44osvIm7IeGqfcaSzNkeHqGqzqobkBrQCXgt43B4Yk8a27XA9qIIZ7bdBgwaaVXPmzDlp2TPPqILq2rVZ3n3YSW7vmjVr9JJLLlFAq1evrlOnTtWkpCR/gwuR1D7jSGdtjg6R2GZgkabyfR/KHtTfwNkBj8t5y/5FRK4CHgMaq+o/IYwnXZ98AnXqQCTOCHHs2DEASpQowd69e3nllVfo1KkT+fLZVQbGmPAVym+ohUBVEamIS0z/Be4I3EBE6gEvA81VdXsIY0lXYiL89BP07OlXBKGxefNmBg8ezNy5c1m5ciUlSpRg+fLlNlzcGJMrhGwUn6omAD2AL4FVwGRVXSEiT4jITd5mI4HCwIciskREpoUqnvT89RccPeqmdo8E+/btY8CAAVSpUoW33nqLmjVrcuTIEQBLTsaYXCOkx3hUdQYwI8WygQH3rwrl6wdr9Wr3t1o1f+PIDsuXL+fKK69k586dtG3blqFDh/LXX39x2mmn+R2aMcZkilWSAJYvd39r1fI3jlOVlJTE+vXrAahevTo33HADixYt4v3336dSpUo+R2eMMafGEhSwYgWceSbkxhkjZs2axYUXXsill17KwYMHyZ8/P2+++SYNGjTwOzRjjMkSG8YFrFwJNVLWuAhzS5Ys4ZFHHuGrr77inHPOYeTIkVaWyBifHDt2jE2bNh0/1xtKRYsWZdWqVSF/nVAoVKgQ5cqVI3/+/EFtH/UJShV++w06dPA7kuAtXbqU+vXrU6xYMUaNGsV9990XcRfaGpObbNq0idjYWCpUqBDygUj79+8nNjY2pK8RCqpKfHw8mzZtomKQ5Xqi/hBffDzs3x/+1z/t2rWLGTPceJPatWszduxY1q9fT69evSw5GeOzI0eOULJkSRslmw4RoWTJkpnqZUZ9glq3zv0N1/p7hw8f5umnn6ZSpUq0adOGffv2ISLce++9FCtWzO/wjDEeS04Zy+x7FPUJ6uuv3d8LL/Q3jpQSExN58803qVatGv369eOyyy7jxx9/tArjxpioEfUJauFCN0DizDP9juTfVq9eTefOnTnzzDOJi4tj+vTp1Mqt4+CNMWHnrbfeokePHgCMHz+eCRMmAPDbb79Rt25d6tWrx7p163j//fd9izHqE9S6dVClit9ROD/99BPDhw8HoEaNGsyfP5/58+fTuHFjnyMzxkSye+65h7vuuguAqVOn0qpVK3755Rc2btzoa4KK6lF8iYmwfj1ce62/caxZs4b+/fszZcoUypQpwz333EOxYsVo2LChv4EZYzLtwQdhyZLs3WfdujB6dPrbbNiwgRtuuIFff/0VgFGjRh2fmqNOnTp8++23JCQk8MYbb5z03TJ48GAKFy5MjRo1GD16NHnz5mXWrFkcPnyYVatWUbduXTp06MBDDz2UvQ3LQFQnqN9+g8OH3Yfvh/j4eAYOHMgrr7xCwYIFGTRoEL169cqVQ0iNMeHr0KFDLFmyhLlz53L33XcfT2IptWjRgnvuuYfChQvTu3dv4uLiGDVqFNOnT8/hiJ2oTlA//+z+1q/vz+snJCTw/vvv06VLFwYNGsQZZ5zhTyDGmGyTUU/HD23btgXgiiuuYN++fezZs8ffgIIU1Qnql18gJgaqV8+Z1zt27BivvvoqX3/9NR9//DFlypThzz//tJF5xpgsy5cvH0lJSccfB15vlHJ4d24ZEh/VgyQWL3aTFObNG9rXUVWmTJlCzZo1ue+++4iPjz/+C8aSkzEmO5QpU4bt27cTHx/PP//886/Dch988AEA3333HUWLFqVo0aJB7TM2Npb9+/eHJN5gRG2COnDAJahQ11T9888/adSoEa1btyZ//vxMmzaNb7/9luLFi4f2hY0xUSV//vwMHDiQhg0bcvXVV1M94NBQoUKFqFevHvfccw+vv/560PusXbs2efPmpU6dOjz33HOhCDtdUXuI79VX4eBBuPPO0Oz/8OHDxMTEUKZMGfLkycNrr71Ghw4dbJp1Y0zI9OzZk54ppgZv0qQJ7dq1Y3SKk2MdO3akY8eOgBvFlyzwfv78+Zk9e3aIos1Y1H5bTp/uRu81apS9+/37778ZNGgQcXFxrFixgkKFCvH999/nmmO+xhgTLqI2QS1dCi1bZt/+9uzZw9NPP83o0aNJTEykR48eHD16lIIFC1pyMsb4Ji4uzu8QTllUJqiDB/MSH599U7yvW7eOhg0bsnv3bu644w6efPLJoMvJG2OMSV1UDpLYsaMgAGeffer7SEpKYsWKFQBUqlSJu+66i59//pl3333XkpMxxmSDqE5QZ511as//+uuvadCgAY0aNWLnzp2ICM899xx1/SpJYYwxESgqE9TmzW5q9AoVMve8n3/+mWuuuYZrrrmGPXv2MG7cOEqUKJH9ARpjjInOc1Dr159O0aJQrlxmnrOeCy64gBIlSvDcc89x7733UrBgwdAFaYwxPqpQoQKLFi2iVKlSvsUQlT2o33+PpU4dyGhw3c6dO49fgV2pUiXeffdd1q5dy4MPPmjJyRgTtlT1X2WPcquo60H98w+sWVOYm29Oe5tDhw7x/PPPM3z4cA4fPkzjxo0544wzuOOOO3IuUGNMrtWkSZOTlt1+++10796dQ4cO0aJFi5PWJ184u3PnTlq1avWvdcEMFd+wYQPXXnstF110EYsXL6Zhw4YsX76cw4cP06pVK4YMGQK4nlGHDh347LPPOHbsGB9++CHVq1cnPj6etm3b8vfff9OoUSNU9fi+n332Wd544w0AunTpwoMPPsiGDRto3rw5F198MT/88AMXXnghnTp1YtCgQWzfvp333nsvy1MGRV0PatUqSEzMQ716J69LSEjgtddeo2rVqvTv35/GjRvzyy+/WJVxY0yusGbNGrp3786KFSt45plnWLRoEcuWLePbb79l2bJlx7crVaoUP//8M/feey+jRo0CYMiQIVx22WWsWLGCli1b8tdffwGwePFi3nzzTRYsWMD8+fN59dVX+eWXXwBYu3YtvXr14rfffuO3337j/fff57vvvmPUqFE89dRTWW5P1PWgkqdBSW329K1bt9KjRw/q1avHxIkTueKKK3I2OGNMREivx3Paaaelu75UqVKnfHHtOeecw8UXXwzA5MmTeeWVV0hISGDLli2sXLmS2rVrA3DrrbcC0KBBAz7++GMA5s6de/z+9ddff7xe6HfffUfLli05/fTTjz933rx53HTTTVSsWJHzzz8fgJo1a9KsWTNEhPPPP58NGzacUhsChbQHJSLNRWS1iKwVkX6prC8oIh946xeISIVQxgOwejXkyaNUreoez58/nz59+qCqlCtXjsWLF/PDDz9YcjLG5DrJSeSPP/5g1KhRzJo1i2XLlnH99df/a/qN5HPoefPmJSEh4ZRfL/BcfJ48eY4/zpMnT5b2e3yfWd5DGkQkLzAWuA6oAbQVkRopNusM7FbVKsBzwNOhiifZ6tVQtuxh/vhjNbfddhuNGjXinXfeYcuWLYD7FWCliYwxudm+ffs4/fTTKVq0KNu2bWPmzJkZPueKK67g/fffB2DmzJns3r0bgMsvv5ypU6dy6NAhDh48yCeffMLll18e0viThfIQX0NgraquBxCRScDNwMqAbW4GBnv3pwBjREQ08OxcNrviit2sWHEfNWtOJiYmhiFDhvDwww9TuHDhUL2kMcbkqDp16lCvXj2qV6/O2WefzaWXXprhcwYNGkTbtm2pWbMml1xyCeXLlwegfv36dOzY8fiAhy5dulCvXr1sOYSXIVUNyQ1oBbwW8Lg9MCbFNr8C5QIerwNKpbffBg0aaFbs27dPS5curffdd59u3bo1S/vKLebMmeN3CDnO2hwdwqXNK1euzLHX2rdvX469Viik9l4BizSV7/tcMUhCRLoB3cDNGpnV6rzjxo2jZMmSrFq1ilWrVmVDhOHtwIEDubqi8amwNkeHcGlz0aJFc2zm2cTERF9nuc2qI0eOBP2ZhTJB/Q0ElmMt5y1LbZtNIpIPKArEp9yRqr4CvAJwwQUXaGrXGGRGXFxcqtcpRKpoay9Ym6NFuLR51apVxMbG5shr7d+/P8deKxSSZ/cNRihH8S0EqopIRREpAPwXmJZim2lAB+9+K2C2190zxphcxb66MpbZ9yhkCUpVE4AewJfAKmCyqq4QkSdE5CZvs9eBkiKyFngYOGkoujHGhLtChQoRHx9vSSodqkp8fDyFChUK+jkhPQelqjOAGSmWDQy4fwRoHcoYjDEm1MqVK8emTZvYsWNHyF/ryJEjmfqSDyeFChWiXCaqdOeKQRLGGBPO8ufPn2MTlcbFxQV9Die3i7pafMYYY3IHS1DGGGPCkiUoY4wxYUly26gTEdkB/JnF3ZQCdmZDOLlFtLUXrM3RwtocGc5R1dIpF+a6BJUdRGSRql7gdxw5JdraC9bmaGFtjmx2iM8YY0xYsgRljDEmLEVrgnrF7wByWLS1F6zN0cLaHMGi8hyUMcaY8BetPShjjDFhzhKUMcaYsBSxCUpEmovIahFZKyInVUkXkYIi8oG3foGIVPAhzGwVRJs7isgOEVni3br4EWd2EZE3RGS7iPyaxnoRkRe892OZiNTP6RizWxBtbiIiewM+44GpbZebiMjZIjJHRFaKyAoReSCVbSLmsw6yvRH3OacqtWl2c/sNyIubPr4SUABYCtRIsU13YLx3/7/AB37HnQNt7giM8TvWbGzzFUB94Nc01rcAZgICXAws8DvmHGhzE2C633Fmc5vLAvW9+7HA76n8246YzzrI9kbc55zaLVJ7UA2Btaq6XlWPApOAm1NsczPwtnd/CtBMRCQHY8xuwbQ5oqjqXGBXOpvcDExQZz5QTETK5kx0oRFEmyOOqm5R1Z+9+/tx88udlWKziPmsg2xvVIjUBHUWsDHg8SZO/oCPb6NucsW9QMkciS40gmkzwG3eIZApInJ2zoTmm2Dfk0jTSESWishMEanpdzDZyTsUXw9YkGJVRH7W6bQXIvhzThapCcqk7jOggqrWBr7mRA/SRI6fcXXN6gAvAlP9DSf7iEhh4CPgQVXd53c8oZZBeyP2cw4UqQnqbyCwd1DOW5bqNiKSDygKxOdIdKGRYZtVNV5V//EevgY0yKHY/BLMv4OIoqr7VPWAd38GkF9ESvkcVpaJSH7cl/V7qvpxKptE1GedUXsj9XNOKVIT1EKgqohUFJECuEEQ01JsMw3o4N1vBcxW7+xjLpVhm1Mck78Jd2w7kk0D7vJGeF0M7FXVLX4HFUoickbyuVQRaYj7P56bf3jhted1YJWqPpvGZhHzWQfT3kj8nFMTkVO+q2qCiPQAvsSNbntDVVeIyBPAIlWdhvsH8I6IrMWddP6vfxFnXZBt7ikiNwEJuDZ39C3gbCAiE3GjmUqJyCZgEJAfQFXHAzNwo7vWAoeATv5Emn2CaHMr4F4RSQAOA//N5T+8AC4F2gPLRWSJt6w/UB4i8rMOpr2R+DmfxEodGWOMCUuReojPGGNMLmcJyhhjTFiyBGWMMSYsWYIyxhgTlixBGWOMOSUZFS9OZfvbA4rgvp/R9pagTNQRERWRZwIe9xaRwdm077dEpFU27WuBV6n6L/l3FfoK2bH/VF6v/yk8p6OIjAlFPCZXeAtoHsyGIlIVeBS4VFVrAg9m9BxLUCYa/QPcGm5X3nsVTY5T1YtUtS4wEFdtv65325CZ/WRCphOUiW6pFS8Wkcoi8oWILBaReSJS3VvVFRirqru9527PaP+WoEw0SgBeAR5KuSJlD0hEDnh/m4jItyLyqYisF5HhInKniPwkIstFpHLAbq4SkUUi8ruI3OA9P6+IjBSRhV6x3v8L2O88EZkGrMwocBG50etZ/SIi34hIGW/5YBF5R0S+x12AXlpEvvYOpbwmIn8mJ2QRaefFvUREXvZiGw7EeMveS2s7b3knr20/4S4qNSbQK8D9qtoA6A285C2vBlQTke9FZL6IZNjzsgRlotVY4E4RKZqJ59QB7gHOw13pX01VG+LqGt4fsF0F3PQn1wPjRaQQ0BlXfudC4EKgq4hU9LavDzygqtWCiOE74GJVrYebUqVvwLoawFWq2hZXYWK2dyhlCl4VAhE5D2iDO8xSF0gE7lTVfsBhr4d2Z1rbiSuXNQSXmC7zXtMY4HiB20uAD70qGC/j5rcCV7moKq4SSlvgVREplt7+IrLUkTEZUdV9IjIB6IkrFROMhcn13URkHfCVt3w50DRgu8mqmgSsEZH1QHXgGqB2QO+sKO4/61HgJ1X9I8gYygEfeImiABD4vGmqmtyWy4CWXlu/EJHd3vJmuCLBC71SbjFAaoda0truIiBOVXd478MHuF/GxoDr9OzxftSktAk3keQx4A8R+R33f2BhejszJlqNxvVsTg9YloD3/0JE8uCSQLJ/Au4nBTxO4t8/9lLWD1PcTK/3B5xHqqiqyQnuYCZifhE3K/L5wP8BhQLWBbMfAd4OiONcVR2che2MOc6bFuQPEWkNrvCtiNTxVk/F9Z7wDjdXA9antz9LUCZqqeouYDIuSSXbwIlpSG7CK8SaSa1FJI93XqoSsBpXxPdecdMoICLVROT09HaShqKcmEaiQzrbfQ/c7r3WNUBxb/ksoJWI/MdbV0JEzvHWHUuOL53tFgCNRaSkt23rU2iDiRDiihf/CJwrIptEpDNwJ9BZRJYCKzgxs/eXQLyIrATmAH1UNd0K7HaIz0S7Z4AeAY9fBT71/nN9QeZ6N8n+An4CigD3qOoREXkNd27qZ3HHzHYAt5zCvgfjju/vBmYDFdPYbggwUUTa475AtgL7VXWniAwAvvJ6iMeA+4A/cSe3l4nIz955qJO2U9X54obk/wjsAZacQhtMhPDOd6bmpAEQXrX1h71bUKyauTERSEQKAoneNCyNgHFpnBcwJmxZD8qYyFQemOz1fo7irkExJlexHpQxxpiwZIMkjDHGhCVLUMYYY8KSJShjjDFhyRKUMcaYsGQJyhhjTFj6f6eXwUYz56EPAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "perf_h2o.plot_uplift(metric=\"gain\", normalize=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8938295747088778"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "perf_h2o.auuc_normalized()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scoring histrory and importance of number of trees\n",
    "\n",
    "To speed up the calculation of AUUC, the predictions are binned into quantile histograms. To calculate precision AUUC the more bins the better. The more trees usually produce more various predictions and then the algorithm creates histograms with more bins. So the algorithm needs more iterations to get meaningful AUUC results. \n",
    "You can see in the scoring history table the number of bins as well as the result AUUC. There is also Qini value parameter, which reflects the number of bins and then is a better pointer of the model improvement. In the scoring history table below you can see the algorithm stabilized after building 6 trees. But it depends on data and model settings on how many trees are necessary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>duration</th>\n",
       "      <th>number_of_trees</th>\n",
       "      <th>training_auuc_nbins</th>\n",
       "      <th>training_auuc</th>\n",
       "      <th>training_auuc_normalized</th>\n",
       "      <th>training_qini_value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td></td>\n",
       "      <td>2023-07-20 15:32:20</td>\n",
       "      <td>0.087 sec</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td></td>\n",
       "      <td>2023-07-20 15:32:25</td>\n",
       "      <td>5.001 sec</td>\n",
       "      <td>1.0</td>\n",
       "      <td>16</td>\n",
       "      <td>27654.759831</td>\n",
       "      <td>0.835491</td>\n",
       "      <td>145.197947</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td></td>\n",
       "      <td>2023-07-20 15:32:41</td>\n",
       "      <td>20.732 sec</td>\n",
       "      <td>2.0</td>\n",
       "      <td>36</td>\n",
       "      <td>24922.986029</td>\n",
       "      <td>0.752960</td>\n",
       "      <td>659.460775</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td></td>\n",
       "      <td>2023-07-20 15:32:56</td>\n",
       "      <td>35.267 sec</td>\n",
       "      <td>3.0</td>\n",
       "      <td>61</td>\n",
       "      <td>23986.477620</td>\n",
       "      <td>0.724667</td>\n",
       "      <td>1427.558101</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td></td>\n",
       "      <td>2023-07-20 15:33:11</td>\n",
       "      <td>51.081 sec</td>\n",
       "      <td>4.0</td>\n",
       "      <td>97</td>\n",
       "      <td>23745.531904</td>\n",
       "      <td>0.717388</td>\n",
       "      <td>2139.213693</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td></td>\n",
       "      <td>2023-07-20 15:33:28</td>\n",
       "      <td>1 min  7.611 sec</td>\n",
       "      <td>5.0</td>\n",
       "      <td>141</td>\n",
       "      <td>23853.273732</td>\n",
       "      <td>0.720643</td>\n",
       "      <td>2742.899584</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td></td>\n",
       "      <td>2023-07-20 15:33:46</td>\n",
       "      <td>1 min 25.335 sec</td>\n",
       "      <td>6.0</td>\n",
       "      <td>208</td>\n",
       "      <td>24297.465037</td>\n",
       "      <td>0.734062</td>\n",
       "      <td>3312.883020</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td></td>\n",
       "      <td>2023-07-20 15:34:04</td>\n",
       "      <td>1 min 43.525 sec</td>\n",
       "      <td>7.0</td>\n",
       "      <td>277</td>\n",
       "      <td>24783.705217</td>\n",
       "      <td>0.748752</td>\n",
       "      <td>3782.834258</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td></td>\n",
       "      <td>2023-07-20 15:34:26</td>\n",
       "      <td>2 min  6.050 sec</td>\n",
       "      <td>8.0</td>\n",
       "      <td>363</td>\n",
       "      <td>25383.216281</td>\n",
       "      <td>0.766865</td>\n",
       "      <td>4234.377179</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td></td>\n",
       "      <td>2023-07-20 15:34:50</td>\n",
       "      <td>2 min 29.392 sec</td>\n",
       "      <td>9.0</td>\n",
       "      <td>455</td>\n",
       "      <td>25490.020313</td>\n",
       "      <td>0.770091</td>\n",
       "      <td>4383.016777</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td></td>\n",
       "      <td>2023-07-20 15:35:14</td>\n",
       "      <td>2 min 53.175 sec</td>\n",
       "      <td>10.0</td>\n",
       "      <td>535</td>\n",
       "      <td>25794.977981</td>\n",
       "      <td>0.779304</td>\n",
       "      <td>4601.770290</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td></td>\n",
       "      <td>2023-07-20 15:35:39</td>\n",
       "      <td>3 min 18.196 sec</td>\n",
       "      <td>11.0</td>\n",
       "      <td>625</td>\n",
       "      <td>26207.635843</td>\n",
       "      <td>0.791771</td>\n",
       "      <td>4852.390299</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td></td>\n",
       "      <td>2023-07-20 15:36:04</td>\n",
       "      <td>3 min 43.319 sec</td>\n",
       "      <td>12.0</td>\n",
       "      <td>702</td>\n",
       "      <td>26590.000175</td>\n",
       "      <td>0.803323</td>\n",
       "      <td>5071.457880</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td></td>\n",
       "      <td>2023-07-20 15:36:30</td>\n",
       "      <td>4 min  9.613 sec</td>\n",
       "      <td>13.0</td>\n",
       "      <td>763</td>\n",
       "      <td>26767.165336</td>\n",
       "      <td>0.808676</td>\n",
       "      <td>5177.909341</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td></td>\n",
       "      <td>2023-07-20 15:36:57</td>\n",
       "      <td>4 min 36.351 sec</td>\n",
       "      <td>14.0</td>\n",
       "      <td>831</td>\n",
       "      <td>27048.766708</td>\n",
       "      <td>0.817183</td>\n",
       "      <td>5333.618998</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td></td>\n",
       "      <td>2023-07-20 15:37:26</td>\n",
       "      <td>5 min  6.015 sec</td>\n",
       "      <td>15.0</td>\n",
       "      <td>881</td>\n",
       "      <td>27155.990512</td>\n",
       "      <td>0.820423</td>\n",
       "      <td>5397.093335</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td></td>\n",
       "      <td>2023-07-20 15:37:59</td>\n",
       "      <td>5 min 39.043 sec</td>\n",
       "      <td>16.0</td>\n",
       "      <td>917</td>\n",
       "      <td>27302.334653</td>\n",
       "      <td>0.824844</td>\n",
       "      <td>5475.851086</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td></td>\n",
       "      <td>2023-07-20 15:38:30</td>\n",
       "      <td>6 min  9.563 sec</td>\n",
       "      <td>17.0</td>\n",
       "      <td>937</td>\n",
       "      <td>27345.073266</td>\n",
       "      <td>0.826135</td>\n",
       "      <td>5501.832671</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td></td>\n",
       "      <td>2023-07-20 15:39:04</td>\n",
       "      <td>6 min 43.276 sec</td>\n",
       "      <td>18.0</td>\n",
       "      <td>955</td>\n",
       "      <td>27482.505135</td>\n",
       "      <td>0.830287</td>\n",
       "      <td>5575.609166</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td></td>\n",
       "      <td>2023-07-20 15:39:38</td>\n",
       "      <td>7 min 17.728 sec</td>\n",
       "      <td>19.0</td>\n",
       "      <td>968</td>\n",
       "      <td>27600.653828</td>\n",
       "      <td>0.833857</td>\n",
       "      <td>5636.695760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td></td>\n",
       "      <td>2023-07-20 15:40:18</td>\n",
       "      <td>7 min 57.455 sec</td>\n",
       "      <td>20.0</td>\n",
       "      <td>975</td>\n",
       "      <td>27653.366952</td>\n",
       "      <td>0.835449</td>\n",
       "      <td>5665.486356</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                timestamp           duration  number_of_trees  \\\n",
       "0     2023-07-20 15:32:20          0.087 sec              0.0   \n",
       "1     2023-07-20 15:32:25          5.001 sec              1.0   \n",
       "2     2023-07-20 15:32:41         20.732 sec              2.0   \n",
       "3     2023-07-20 15:32:56         35.267 sec              3.0   \n",
       "4     2023-07-20 15:33:11         51.081 sec              4.0   \n",
       "5     2023-07-20 15:33:28   1 min  7.611 sec              5.0   \n",
       "6     2023-07-20 15:33:46   1 min 25.335 sec              6.0   \n",
       "7     2023-07-20 15:34:04   1 min 43.525 sec              7.0   \n",
       "8     2023-07-20 15:34:26   2 min  6.050 sec              8.0   \n",
       "9     2023-07-20 15:34:50   2 min 29.392 sec              9.0   \n",
       "10    2023-07-20 15:35:14   2 min 53.175 sec             10.0   \n",
       "11    2023-07-20 15:35:39   3 min 18.196 sec             11.0   \n",
       "12    2023-07-20 15:36:04   3 min 43.319 sec             12.0   \n",
       "13    2023-07-20 15:36:30   4 min  9.613 sec             13.0   \n",
       "14    2023-07-20 15:36:57   4 min 36.351 sec             14.0   \n",
       "15    2023-07-20 15:37:26   5 min  6.015 sec             15.0   \n",
       "16    2023-07-20 15:37:59   5 min 39.043 sec             16.0   \n",
       "17    2023-07-20 15:38:30   6 min  9.563 sec             17.0   \n",
       "18    2023-07-20 15:39:04   6 min 43.276 sec             18.0   \n",
       "19    2023-07-20 15:39:38   7 min 17.728 sec             19.0   \n",
       "20    2023-07-20 15:40:18   7 min 57.455 sec             20.0   \n",
       "\n",
       "    training_auuc_nbins  training_auuc  training_auuc_normalized  \\\n",
       "0                     0            NaN                       NaN   \n",
       "1                    16   27654.759831                  0.835491   \n",
       "2                    36   24922.986029                  0.752960   \n",
       "3                    61   23986.477620                  0.724667   \n",
       "4                    97   23745.531904                  0.717388   \n",
       "5                   141   23853.273732                  0.720643   \n",
       "6                   208   24297.465037                  0.734062   \n",
       "7                   277   24783.705217                  0.748752   \n",
       "8                   363   25383.216281                  0.766865   \n",
       "9                   455   25490.020313                  0.770091   \n",
       "10                  535   25794.977981                  0.779304   \n",
       "11                  625   26207.635843                  0.791771   \n",
       "12                  702   26590.000175                  0.803323   \n",
       "13                  763   26767.165336                  0.808676   \n",
       "14                  831   27048.766708                  0.817183   \n",
       "15                  881   27155.990512                  0.820423   \n",
       "16                  917   27302.334653                  0.824844   \n",
       "17                  937   27345.073266                  0.826135   \n",
       "18                  955   27482.505135                  0.830287   \n",
       "19                  968   27600.653828                  0.833857   \n",
       "20                  975   27653.366952                  0.835449   \n",
       "\n",
       "    training_qini_value  \n",
       "0                   NaN  \n",
       "1            145.197947  \n",
       "2            659.460775  \n",
       "3           1427.558101  \n",
       "4           2139.213693  \n",
       "5           2742.899584  \n",
       "6           3312.883020  \n",
       "7           3782.834258  \n",
       "8           4234.377179  \n",
       "9           4383.016777  \n",
       "10          4601.770290  \n",
       "11          4852.390299  \n",
       "12          5071.457880  \n",
       "13          5177.909341  \n",
       "14          5333.618998  \n",
       "15          5397.093335  \n",
       "16          5475.851086  \n",
       "17          5501.832671  \n",
       "18          5575.609166  \n",
       "19          5636.695760  \n",
       "20          5665.486356  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "h2o_uplift_model.scoring_history()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparasion Tree-based approach and Generalized Weighed Uplift (LGWUM)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "LGWUM (Kane et al., 2014) is one of several methods available for Uplift Modeling, and uses an approach to Uplift Modelling better known as Class Variable Transformation. LGWUM assumes that positive uplift lies in treating treatment-group responders (TR) and control-group non-responders (CN), whilst avoiding treatment-group non-responders (TN) and control-group responders (CR). This is visually shown as:\n",
    "\n",
    "𝑈𝑝𝑙𝑖𝑓𝑡 𝐿𝐺𝑊𝑈𝑀 = P(TR)/P(T) + P(CN)/P(C) - P(TN)/P(T) - P(CR)/P(C)\n",
    "\n",
    "source: https://www.kaggle.com/code/hughhuyton/criteo-uplift-modelling/notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gbm Model Build progress: |██████████████████████████████████████████████████████| (done) 100%\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style='margin: 1em 0 1em 0;'>Model Details\n",
       "=============\n",
       "H2OGradientBoostingEstimator : Gradient Boosting Machine\n",
       "Model Key: GBM_model_python_1689859822235_24\n",
       "</pre>\n",
       "<div style='margin: 1em 0 1em 0;'>\n",
       "<style>\n",
       "\n",
       "#h2o-table-8.h2o-container {\n",
       "  overflow-x: auto;\n",
       "}\n",
       "#h2o-table-8 .h2o-table {\n",
       "  /* width: 100%; */\n",
       "  margin-top: 1em;\n",
       "  margin-bottom: 1em;\n",
       "}\n",
       "#h2o-table-8 .h2o-table caption {\n",
       "  white-space: nowrap;\n",
       "  caption-side: top;\n",
       "  text-align: left;\n",
       "  /* margin-left: 1em; */\n",
       "  margin: 0;\n",
       "  font-size: larger;\n",
       "}\n",
       "#h2o-table-8 .h2o-table thead {\n",
       "  white-space: nowrap; \n",
       "  position: sticky;\n",
       "  top: 0;\n",
       "  box-shadow: 0 -1px inset;\n",
       "}\n",
       "#h2o-table-8 .h2o-table tbody {\n",
       "  overflow: auto;\n",
       "}\n",
       "#h2o-table-8 .h2o-table th,\n",
       "#h2o-table-8 .h2o-table td {\n",
       "  text-align: right;\n",
       "  /* border: 1px solid; */\n",
       "}\n",
       "#h2o-table-8 .h2o-table tr:nth-child(even) {\n",
       "  /* background: #F5F5F5 */\n",
       "}\n",
       "\n",
       "</style>      \n",
       "<div id=\"h2o-table-8\" class=\"h2o-container\">\n",
       "  <table class=\"h2o-table\">\n",
       "    <caption>Model Summary: </caption>\n",
       "    <thead><tr><th></th>\n",
       "<th>number_of_trees</th>\n",
       "<th>number_of_internal_trees</th>\n",
       "<th>model_size_in_bytes</th>\n",
       "<th>min_depth</th>\n",
       "<th>max_depth</th>\n",
       "<th>mean_depth</th>\n",
       "<th>min_leaves</th>\n",
       "<th>max_leaves</th>\n",
       "<th>mean_leaves</th></tr></thead>\n",
       "    <tbody><tr><td></td>\n",
       "<td>20.0</td>\n",
       "<td>80.0</td>\n",
       "<td>3989386.0</td>\n",
       "<td>15.0</td>\n",
       "<td>15.0</td>\n",
       "<td>15.0</td>\n",
       "<td>1336.0</td>\n",
       "<td>7040.0</td>\n",
       "<td>3969.325</td></tr></tbody>\n",
       "  </table>\n",
       "</div>\n",
       "</div>\n",
       "<div style='margin: 1em 0 1em 0;'><pre style='margin: 1em 0 1em 0;'>ModelMetricsMultinomial: gbm\n",
       "** Reported on train data. **\n",
       "\n",
       "MSE: 0.31190785343648125\n",
       "RMSE: 0.5584871112536808\n",
       "LogLoss: 0.8285843718112937\n",
       "Mean Per-Class Error: 0.5822510845469803\n",
       "AUC table was not computed: it is either disabled (model parameter 'auc_type' was set to AUTO or NONE) or the domain size exceeds the limit (maximum is 50 domains).\n",
       "AUCPR table was not computed: it is either disabled (model parameter 'auc_type' was set to AUTO or NONE) or the domain size exceeds the limit (maximum is 50 domains).</pre>\n",
       "<div style='margin: 1em 0 1em 0;'>\n",
       "<style>\n",
       "\n",
       "#h2o-table-9.h2o-container {\n",
       "  overflow-x: auto;\n",
       "}\n",
       "#h2o-table-9 .h2o-table {\n",
       "  /* width: 100%; */\n",
       "  margin-top: 1em;\n",
       "  margin-bottom: 1em;\n",
       "}\n",
       "#h2o-table-9 .h2o-table caption {\n",
       "  white-space: nowrap;\n",
       "  caption-side: top;\n",
       "  text-align: left;\n",
       "  /* margin-left: 1em; */\n",
       "  margin: 0;\n",
       "  font-size: larger;\n",
       "}\n",
       "#h2o-table-9 .h2o-table thead {\n",
       "  white-space: nowrap; \n",
       "  position: sticky;\n",
       "  top: 0;\n",
       "  box-shadow: 0 -1px inset;\n",
       "}\n",
       "#h2o-table-9 .h2o-table tbody {\n",
       "  overflow: auto;\n",
       "}\n",
       "#h2o-table-9 .h2o-table th,\n",
       "#h2o-table-9 .h2o-table td {\n",
       "  text-align: right;\n",
       "  /* border: 1px solid; */\n",
       "}\n",
       "#h2o-table-9 .h2o-table tr:nth-child(even) {\n",
       "  /* background: #F5F5F5 */\n",
       "}\n",
       "\n",
       "</style>      \n",
       "<div id=\"h2o-table-9\" class=\"h2o-container\">\n",
       "  <table class=\"h2o-table\">\n",
       "    <caption>Confusion Matrix: Row labels: Actual class; Column labels: Predicted class</caption>\n",
       "    <thead><tr><th>0</th>\n",
       "<th>1</th>\n",
       "<th>2</th>\n",
       "<th>3</th>\n",
       "<th>Error</th>\n",
       "<th>Rate</th></tr></thead>\n",
       "    <tbody><tr><td>1471382.0</td>\n",
       "<td>3497.0</td>\n",
       "<td>130706.0</td>\n",
       "<td>7860.0</td>\n",
       "<td>0.0880495</td>\n",
       "<td>142,063 / 1,613,445</td></tr>\n",
       "<tr><td>23449.0</td>\n",
       "<td>14885.0</td>\n",
       "<td>17613.0</td>\n",
       "<td>8158.0</td>\n",
       "<td>0.7678028</td>\n",
       "<td>49,220 / 64,105</td></tr>\n",
       "<tr><td>1329149.0</td>\n",
       "<td>3522.0</td>\n",
       "<td>256258.0</td>\n",
       "<td>7966.0</td>\n",
       "<td>0.8395273</td>\n",
       "<td>1,340,637 / 1,596,895</td></tr>\n",
       "<tr><td>26155.0</td>\n",
       "<td>2917.0</td>\n",
       "<td>22033.0</td>\n",
       "<td>29550.0</td>\n",
       "<td>0.6336247</td>\n",
       "<td>51,105 / 80,655</td></tr>\n",
       "<tr><td>2850135.0</td>\n",
       "<td>24821.0</td>\n",
       "<td>426610.0</td>\n",
       "<td>53534.0</td>\n",
       "<td>0.4718265</td>\n",
       "<td>1,583,025 / 3,355,100</td></tr></tbody>\n",
       "  </table>\n",
       "</div>\n",
       "</div>\n",
       "<div style='margin: 1em 0 1em 0;'>\n",
       "<style>\n",
       "\n",
       "#h2o-table-10.h2o-container {\n",
       "  overflow-x: auto;\n",
       "}\n",
       "#h2o-table-10 .h2o-table {\n",
       "  /* width: 100%; */\n",
       "  margin-top: 1em;\n",
       "  margin-bottom: 1em;\n",
       "}\n",
       "#h2o-table-10 .h2o-table caption {\n",
       "  white-space: nowrap;\n",
       "  caption-side: top;\n",
       "  text-align: left;\n",
       "  /* margin-left: 1em; */\n",
       "  margin: 0;\n",
       "  font-size: larger;\n",
       "}\n",
       "#h2o-table-10 .h2o-table thead {\n",
       "  white-space: nowrap; \n",
       "  position: sticky;\n",
       "  top: 0;\n",
       "  box-shadow: 0 -1px inset;\n",
       "}\n",
       "#h2o-table-10 .h2o-table tbody {\n",
       "  overflow: auto;\n",
       "}\n",
       "#h2o-table-10 .h2o-table th,\n",
       "#h2o-table-10 .h2o-table td {\n",
       "  text-align: right;\n",
       "  /* border: 1px solid; */\n",
       "}\n",
       "#h2o-table-10 .h2o-table tr:nth-child(even) {\n",
       "  /* background: #F5F5F5 */\n",
       "}\n",
       "\n",
       "</style>      \n",
       "<div id=\"h2o-table-10\" class=\"h2o-container\">\n",
       "  <table class=\"h2o-table\">\n",
       "    <caption>Top-4 Hit Ratios: </caption>\n",
       "    <thead><tr><th>k</th>\n",
       "<th>hit_ratio</th></tr></thead>\n",
       "    <tbody><tr><td>1</td>\n",
       "<td>0.5281735</td></tr>\n",
       "<tr><td>2</td>\n",
       "<td>0.9707624</td></tr>\n",
       "<tr><td>3</td>\n",
       "<td>0.9937637</td></tr>\n",
       "<tr><td>4</td>\n",
       "<td>0.9999999</td></tr></tbody>\n",
       "  </table>\n",
       "</div>\n",
       "</div></div>\n",
       "<div style='margin: 1em 0 1em 0;'>\n",
       "<style>\n",
       "\n",
       "#h2o-table-11.h2o-container {\n",
       "  overflow-x: auto;\n",
       "}\n",
       "#h2o-table-11 .h2o-table {\n",
       "  /* width: 100%; */\n",
       "  margin-top: 1em;\n",
       "  margin-bottom: 1em;\n",
       "}\n",
       "#h2o-table-11 .h2o-table caption {\n",
       "  white-space: nowrap;\n",
       "  caption-side: top;\n",
       "  text-align: left;\n",
       "  /* margin-left: 1em; */\n",
       "  margin: 0;\n",
       "  font-size: larger;\n",
       "}\n",
       "#h2o-table-11 .h2o-table thead {\n",
       "  white-space: nowrap; \n",
       "  position: sticky;\n",
       "  top: 0;\n",
       "  box-shadow: 0 -1px inset;\n",
       "}\n",
       "#h2o-table-11 .h2o-table tbody {\n",
       "  overflow: auto;\n",
       "}\n",
       "#h2o-table-11 .h2o-table th,\n",
       "#h2o-table-11 .h2o-table td {\n",
       "  text-align: right;\n",
       "  /* border: 1px solid; */\n",
       "}\n",
       "#h2o-table-11 .h2o-table tr:nth-child(even) {\n",
       "  /* background: #F5F5F5 */\n",
       "}\n",
       "\n",
       "</style>      \n",
       "<div id=\"h2o-table-11\" class=\"h2o-container\">\n",
       "  <table class=\"h2o-table\">\n",
       "    <caption>Scoring History: </caption>\n",
       "    <thead><tr><th></th>\n",
       "<th>timestamp</th>\n",
       "<th>duration</th>\n",
       "<th>number_of_trees</th>\n",
       "<th>training_rmse</th>\n",
       "<th>training_logloss</th>\n",
       "<th>training_classification_error</th>\n",
       "<th>training_auc</th>\n",
       "<th>training_pr_auc</th></tr></thead>\n",
       "    <tbody><tr><td></td>\n",
       "<td>2023-07-20 15:41:27</td>\n",
       "<td> 0.029 sec</td>\n",
       "<td>0.0</td>\n",
       "<td>0.75</td>\n",
       "<td>1.3862944</td>\n",
       "<td>0.5408468</td>\n",
       "<td>nan</td>\n",
       "<td>nan</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:41:43</td>\n",
       "<td>16.020 sec</td>\n",
       "<td>1.0</td>\n",
       "<td>0.7273500</td>\n",
       "<td>1.2998644</td>\n",
       "<td>0.4976025</td>\n",
       "<td>nan</td>\n",
       "<td>nan</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:41:58</td>\n",
       "<td>31.267 sec</td>\n",
       "<td>2.0</td>\n",
       "<td>0.7072115</td>\n",
       "<td>1.2292087</td>\n",
       "<td>0.4930673</td>\n",
       "<td>nan</td>\n",
       "<td>nan</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:42:13</td>\n",
       "<td>45.968 sec</td>\n",
       "<td>3.0</td>\n",
       "<td>0.6892735</td>\n",
       "<td>1.1704853</td>\n",
       "<td>0.4898101</td>\n",
       "<td>nan</td>\n",
       "<td>nan</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:42:27</td>\n",
       "<td>59.670 sec</td>\n",
       "<td>4.0</td>\n",
       "<td>0.6733152</td>\n",
       "<td>1.1211795</td>\n",
       "<td>0.4879554</td>\n",
       "<td>nan</td>\n",
       "<td>nan</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:42:40</td>\n",
       "<td> 1 min 13.559 sec</td>\n",
       "<td>5.0</td>\n",
       "<td>0.6590825</td>\n",
       "<td>1.0793036</td>\n",
       "<td>0.4860174</td>\n",
       "<td>nan</td>\n",
       "<td>nan</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:42:54</td>\n",
       "<td> 1 min 26.651 sec</td>\n",
       "<td>6.0</td>\n",
       "<td>0.6463788</td>\n",
       "<td>1.0434638</td>\n",
       "<td>0.4845158</td>\n",
       "<td>nan</td>\n",
       "<td>nan</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:43:06</td>\n",
       "<td> 1 min 38.976 sec</td>\n",
       "<td>7.0</td>\n",
       "<td>0.6350349</td>\n",
       "<td>1.0126010</td>\n",
       "<td>0.4831436</td>\n",
       "<td>nan</td>\n",
       "<td>nan</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:43:17</td>\n",
       "<td> 1 min 50.555 sec</td>\n",
       "<td>8.0</td>\n",
       "<td>0.6248829</td>\n",
       "<td>0.9858407</td>\n",
       "<td>0.4819573</td>\n",
       "<td>nan</td>\n",
       "<td>nan</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:43:31</td>\n",
       "<td> 2 min  4.354 sec</td>\n",
       "<td>9.0</td>\n",
       "<td>0.6157754</td>\n",
       "<td>0.9624885</td>\n",
       "<td>0.4808304</td>\n",
       "<td>nan</td>\n",
       "<td>nan</td></tr>\n",
       "<tr><td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:43:53</td>\n",
       "<td> 2 min 26.563 sec</td>\n",
       "<td>11.0</td>\n",
       "<td>0.6001792</td>\n",
       "<td>0.9238657</td>\n",
       "<td>0.4783157</td>\n",
       "<td>nan</td>\n",
       "<td>nan</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:44:04</td>\n",
       "<td> 2 min 37.405 sec</td>\n",
       "<td>12.0</td>\n",
       "<td>0.5935326</td>\n",
       "<td>0.9079134</td>\n",
       "<td>0.4772913</td>\n",
       "<td>nan</td>\n",
       "<td>nan</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:44:15</td>\n",
       "<td> 2 min 48.115 sec</td>\n",
       "<td>13.0</td>\n",
       "<td>0.5875160</td>\n",
       "<td>0.8937238</td>\n",
       "<td>0.4763617</td>\n",
       "<td>nan</td>\n",
       "<td>nan</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:44:25</td>\n",
       "<td> 2 min 58.417 sec</td>\n",
       "<td>14.0</td>\n",
       "<td>0.5820820</td>\n",
       "<td>0.8811144</td>\n",
       "<td>0.4754955</td>\n",
       "<td>nan</td>\n",
       "<td>nan</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:44:35</td>\n",
       "<td> 3 min  8.642 sec</td>\n",
       "<td>15.0</td>\n",
       "<td>0.5771645</td>\n",
       "<td>0.8698723</td>\n",
       "<td>0.4748529</td>\n",
       "<td>nan</td>\n",
       "<td>nan</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:44:45</td>\n",
       "<td> 3 min 17.780 sec</td>\n",
       "<td>16.0</td>\n",
       "<td>0.5726998</td>\n",
       "<td>0.8597958</td>\n",
       "<td>0.4742571</td>\n",
       "<td>nan</td>\n",
       "<td>nan</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:44:54</td>\n",
       "<td> 3 min 26.782 sec</td>\n",
       "<td>17.0</td>\n",
       "<td>0.5686278</td>\n",
       "<td>0.8507211</td>\n",
       "<td>0.4735725</td>\n",
       "<td>nan</td>\n",
       "<td>nan</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:45:02</td>\n",
       "<td> 3 min 35.351 sec</td>\n",
       "<td>18.0</td>\n",
       "<td>0.5649397</td>\n",
       "<td>0.8425981</td>\n",
       "<td>0.4730443</td>\n",
       "<td>nan</td>\n",
       "<td>nan</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:45:11</td>\n",
       "<td> 3 min 44.142 sec</td>\n",
       "<td>19.0</td>\n",
       "<td>0.5615733</td>\n",
       "<td>0.8352602</td>\n",
       "<td>0.4724235</td>\n",
       "<td>nan</td>\n",
       "<td>nan</td></tr>\n",
       "<tr><td></td>\n",
       "<td>2023-07-20 15:45:20</td>\n",
       "<td> 3 min 53.009 sec</td>\n",
       "<td>20.0</td>\n",
       "<td>0.5584871</td>\n",
       "<td>0.8285844</td>\n",
       "<td>0.4718265</td>\n",
       "<td>nan</td>\n",
       "<td>nan</td></tr></tbody>\n",
       "  </table>\n",
       "</div>\n",
       "<pre style='font-size: smaller; margin-bottom: 1em;'>[21 rows x 9 columns]</pre></div>\n",
       "<div style='margin: 1em 0 1em 0;'>\n",
       "<style>\n",
       "\n",
       "#h2o-table-12.h2o-container {\n",
       "  overflow-x: auto;\n",
       "}\n",
       "#h2o-table-12 .h2o-table {\n",
       "  /* width: 100%; */\n",
       "  margin-top: 1em;\n",
       "  margin-bottom: 1em;\n",
       "}\n",
       "#h2o-table-12 .h2o-table caption {\n",
       "  white-space: nowrap;\n",
       "  caption-side: top;\n",
       "  text-align: left;\n",
       "  /* margin-left: 1em; */\n",
       "  margin: 0;\n",
       "  font-size: larger;\n",
       "}\n",
       "#h2o-table-12 .h2o-table thead {\n",
       "  white-space: nowrap; \n",
       "  position: sticky;\n",
       "  top: 0;\n",
       "  box-shadow: 0 -1px inset;\n",
       "}\n",
       "#h2o-table-12 .h2o-table tbody {\n",
       "  overflow: auto;\n",
       "}\n",
       "#h2o-table-12 .h2o-table th,\n",
       "#h2o-table-12 .h2o-table td {\n",
       "  text-align: right;\n",
       "  /* border: 1px solid; */\n",
       "}\n",
       "#h2o-table-12 .h2o-table tr:nth-child(even) {\n",
       "  /* background: #F5F5F5 */\n",
       "}\n",
       "\n",
       "</style>      \n",
       "<div id=\"h2o-table-12\" class=\"h2o-container\">\n",
       "  <table class=\"h2o-table\">\n",
       "    <caption>Variable Importances: </caption>\n",
       "    <thead><tr><th>variable</th>\n",
       "<th>relative_importance</th>\n",
       "<th>scaled_importance</th>\n",
       "<th>percentage</th></tr></thead>\n",
       "    <tbody><tr><td>f8</td>\n",
       "<td>185976.1406250</td>\n",
       "<td>1.0</td>\n",
       "<td>0.4403955</td></tr>\n",
       "<tr><td>f2</td>\n",
       "<td>100019.3515625</td>\n",
       "<td>0.5378074</td>\n",
       "<td>0.2368480</td></tr>\n",
       "<tr><td>f9</td>\n",
       "<td>35638.8125000</td>\n",
       "<td>0.1916311</td>\n",
       "<td>0.0843935</td></tr>\n",
       "<tr><td>f0</td>\n",
       "<td>33771.0078125</td>\n",
       "<td>0.1815879</td>\n",
       "<td>0.0799705</td></tr>\n",
       "<tr><td>f6</td>\n",
       "<td>22928.4726562</td>\n",
       "<td>0.1232872</td>\n",
       "<td>0.0542951</td></tr>\n",
       "<tr><td>f3</td>\n",
       "<td>14589.6035156</td>\n",
       "<td>0.0784488</td>\n",
       "<td>0.0345485</td></tr>\n",
       "<tr><td>f10</td>\n",
       "<td>12675.4287109</td>\n",
       "<td>0.0681562</td>\n",
       "<td>0.0300157</td></tr>\n",
       "<tr><td>f7</td>\n",
       "<td>8803.5302734</td>\n",
       "<td>0.0473369</td>\n",
       "<td>0.0208469</td></tr>\n",
       "<tr><td>f4</td>\n",
       "<td>3712.6965332</td>\n",
       "<td>0.0199633</td>\n",
       "<td>0.0087917</td></tr>\n",
       "<tr><td>f5</td>\n",
       "<td>1944.6453857</td>\n",
       "<td>0.0104564</td>\n",
       "<td>0.0046050</td></tr>\n",
       "<tr><td>f11</td>\n",
       "<td>1426.2990723</td>\n",
       "<td>0.0076693</td>\n",
       "<td>0.0033775</td></tr>\n",
       "<tr><td>f1</td>\n",
       "<td>807.4884644</td>\n",
       "<td>0.0043419</td>\n",
       "<td>0.0019121</td></tr></tbody>\n",
       "  </table>\n",
       "</div>\n",
       "</div><pre style=\"font-size: smaller; margin: 1em 0 0 0;\">\n",
       "\n",
       "[tips]\n",
       "Use `model.explain()` to inspect the model.\n",
       "--\n",
       "Use `h2o.display.toggle_user_tips()` to switch on/off this section.</pre>"
      ],
      "text/plain": [
       "Model Details\n",
       "=============\n",
       "H2OGradientBoostingEstimator : Gradient Boosting Machine\n",
       "Model Key: GBM_model_python_1689859822235_24\n",
       "\n",
       "\n",
       "Model Summary: \n",
       "    number_of_trees    number_of_internal_trees    model_size_in_bytes    min_depth    max_depth    mean_depth    min_leaves    max_leaves    mean_leaves\n",
       "--  -----------------  --------------------------  ---------------------  -----------  -----------  ------------  ------------  ------------  -------------\n",
       "    20                 80                          3.98939e+06            15           15           15            1336          7040          3969.32\n",
       "\n",
       "ModelMetricsMultinomial: gbm\n",
       "** Reported on train data. **\n",
       "\n",
       "MSE: 0.31190785343648125\n",
       "RMSE: 0.5584871112536808\n",
       "LogLoss: 0.8285843718112937\n",
       "Mean Per-Class Error: 0.5822510845469803\n",
       "AUC table was not computed: it is either disabled (model parameter 'auc_type' was set to AUTO or NONE) or the domain size exceeds the limit (maximum is 50 domains).\n",
       "AUCPR table was not computed: it is either disabled (model parameter 'auc_type' was set to AUTO or NONE) or the domain size exceeds the limit (maximum is 50 domains).\n",
       "\n",
       "Confusion Matrix: Row labels: Actual class; Column labels: Predicted class\n",
       "0            1      2       3      Error      Rate\n",
       "-----------  -----  ------  -----  ---------  ---------------------\n",
       "1.47138e+06  3497   130706  7860   0.0880495  142,063 / 1,613,445\n",
       "23449        14885  17613   8158   0.767803   49,220 / 64,105\n",
       "1.32915e+06  3522   256258  7966   0.839527   1,340,637 / 1,596,895\n",
       "26155        2917   22033   29550  0.633625   51,105 / 80,655\n",
       "2.85014e+06  24821  426610  53534  0.471826   1,583,025 / 3,355,100\n",
       "\n",
       "Top-4 Hit Ratios: \n",
       "k    hit_ratio\n",
       "---  -----------\n",
       "1    0.528173\n",
       "2    0.970762\n",
       "3    0.993764\n",
       "4    1\n",
       "\n",
       "Scoring History: \n",
       "     timestamp            duration          number_of_trees    training_rmse       training_logloss    training_classification_error    training_auc    training_pr_auc\n",
       "---  -------------------  ----------------  -----------------  ------------------  ------------------  -------------------------------  --------------  -----------------\n",
       "     2023-07-20 15:41:27  0.029 sec         0.0                0.75                1.3862943611201621  0.54084677058806                 nan             nan\n",
       "     2023-07-20 15:41:43  16.020 sec        1.0                0.7273500276934312  1.2998643998347224  0.49760245596256447              nan             nan\n",
       "     2023-07-20 15:41:58  31.267 sec        2.0                0.7072114503050211  1.2292086519554999  0.49306727072218415              nan             nan\n",
       "     2023-07-20 15:42:13  45.968 sec        3.0                0.6892734700013863  1.1704852612632415  0.4898101397871896               nan             nan\n",
       "     2023-07-20 15:42:27  59.670 sec        4.0                0.6733152317145654  1.121179483747584   0.4879553515543501               nan             nan\n",
       "     2023-07-20 15:42:40  1 min 13.559 sec  5.0                0.6590825290094642  1.0793035991503663  0.4860174063366219               nan             nan\n",
       "     2023-07-20 15:42:54  1 min 26.651 sec  6.0                0.6463787602453364  1.0434637681914327  0.4845158117492772               nan             nan\n",
       "     2023-07-20 15:43:06  1 min 38.976 sec  7.0                0.6350349355342839  1.0126009508204046  0.48314357247175943              nan             nan\n",
       "     2023-07-20 15:43:17  1 min 50.555 sec  8.0                0.6248829282266323  0.9858406572017421  0.48195731870883135              nan             nan\n",
       "     2023-07-20 15:43:31  2 min  4.354 sec  9.0                0.6157753561494564  0.9624884977231626  0.4808303776340497               nan             nan\n",
       "---  ---                  ---               ---                ---                 ---                 ---                              ---             ---\n",
       "     2023-07-20 15:43:53  2 min 26.563 sec  11.0               0.6001792474804977  0.9238656584368838  0.4783156984888677               nan             nan\n",
       "     2023-07-20 15:44:04  2 min 37.405 sec  12.0               0.593532620989269   0.9079133765409396  0.4772912878900778               nan             nan\n",
       "     2023-07-20 15:44:15  2 min 48.115 sec  13.0               0.5875160115402669  0.8937237874435485  0.47636165837083844              nan             nan\n",
       "     2023-07-20 15:44:25  2 min 58.417 sec  14.0               0.5820820015034956  0.8811143940935656  0.47549551429167536              nan             nan\n",
       "     2023-07-20 15:44:35  3 min  8.642 sec  15.0               0.5771645132798496  0.8698723044713469  0.4748529104944711               nan             nan\n",
       "     2023-07-20 15:44:45  3 min 17.780 sec  16.0               0.5726998322464905  0.8597958037412026  0.47425710112962355              nan             nan\n",
       "     2023-07-20 15:44:54  3 min 26.782 sec  17.0               0.5686278121224454  0.8507210755533928  0.47357247175941103              nan             nan\n",
       "     2023-07-20 15:45:02  3 min 35.351 sec  18.0               0.5649397025822357  0.8425980575679954  0.4730443205865697               nan             nan\n",
       "     2023-07-20 15:45:11  3 min 44.142 sec  19.0               0.5615733278837302  0.8352602133343008  0.4724234747101428               nan             nan\n",
       "     2023-07-20 15:45:20  3 min 53.009 sec  20.0               0.5584871112536808  0.8285843718112937  0.4718264731304581               nan             nan\n",
       "[21 rows x 9 columns]\n",
       "\n",
       "\n",
       "Variable Importances: \n",
       "variable    relative_importance    scaled_importance    percentage\n",
       "----------  ---------------------  -------------------  ------------\n",
       "f8          185976                 1                    0.440395\n",
       "f2          100019                 0.537807             0.236848\n",
       "f9          35638.8                0.191631             0.0843935\n",
       "f0          33771                  0.181588             0.0799705\n",
       "f6          22928.5                0.123287             0.0542951\n",
       "f3          14589.6                0.0784488            0.0345485\n",
       "f10         12675.4                0.0681562            0.0300157\n",
       "f7          8803.53                0.0473369            0.0208469\n",
       "f4          3712.7                 0.0199633            0.00879174\n",
       "f5          1944.65                0.0104564            0.00460496\n",
       "f11         1426.3                 0.00766926           0.00337751\n",
       "f1          807.488                0.00434189           0.00191215\n",
       "\n",
       "[tips]\n",
       "Use `model.explain()` to inspect the model.\n",
       "--\n",
       "Use `h2o.display.toggle_user_tips()` to switch on/off this section."
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from h2o.estimators.gbm import H2OGradientBoostingEstimator\n",
    "\n",
    "h2o_gbm_lgwum = H2OGradientBoostingEstimator(ntrees=ntree,\n",
    "                                       max_depth=max_depth,\n",
    "                                       min_rows=30,\n",
    "                                       nbins=1000,\n",
    "                                       score_each_iteration=False,\n",
    "                                       seed=42)\n",
    "\n",
    "h2o_gbm_lgwum.train(y=response_column_lgwum, x=feature_cols, training_frame=h2o_train_df)\n",
    "h2o_gbm_lgwum"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gbm prediction progress: |███████████████████████████████████████████████████████| (done) 100%\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>predict</th>\n",
       "      <th>p_cn</th>\n",
       "      <th>p_cr</th>\n",
       "      <th>p_tn</th>\n",
       "      <th>p_tr</th>\n",
       "      <th>uplift_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0.462914</td>\n",
       "      <td>0.039050</td>\n",
       "      <td>0.459005</td>\n",
       "      <td>0.039032</td>\n",
       "      <td>0.001154</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>0.459171</td>\n",
       "      <td>0.040068</td>\n",
       "      <td>0.460331</td>\n",
       "      <td>0.040430</td>\n",
       "      <td>0.000960</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0.461210</td>\n",
       "      <td>0.039238</td>\n",
       "      <td>0.459893</td>\n",
       "      <td>0.039659</td>\n",
       "      <td>0.001968</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>0.451050</td>\n",
       "      <td>0.043022</td>\n",
       "      <td>0.451416</td>\n",
       "      <td>0.054511</td>\n",
       "      <td>0.041337</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0.462915</td>\n",
       "      <td>0.039050</td>\n",
       "      <td>0.459006</td>\n",
       "      <td>0.039029</td>\n",
       "      <td>0.001144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2795914</th>\n",
       "      <td>2</td>\n",
       "      <td>0.262453</td>\n",
       "      <td>0.210295</td>\n",
       "      <td>0.291141</td>\n",
       "      <td>0.236111</td>\n",
       "      <td>0.005958</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2795915</th>\n",
       "      <td>0</td>\n",
       "      <td>0.490065</td>\n",
       "      <td>0.081051</td>\n",
       "      <td>0.337942</td>\n",
       "      <td>0.090942</td>\n",
       "      <td>0.140252</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2795916</th>\n",
       "      <td>0</td>\n",
       "      <td>0.463028</td>\n",
       "      <td>0.039023</td>\n",
       "      <td>0.458692</td>\n",
       "      <td>0.039257</td>\n",
       "      <td>0.002219</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2795917</th>\n",
       "      <td>2</td>\n",
       "      <td>0.448384</td>\n",
       "      <td>0.051104</td>\n",
       "      <td>0.450981</td>\n",
       "      <td>0.049531</td>\n",
       "      <td>-0.006707</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2795918</th>\n",
       "      <td>0</td>\n",
       "      <td>0.462878</td>\n",
       "      <td>0.039047</td>\n",
       "      <td>0.458969</td>\n",
       "      <td>0.039106</td>\n",
       "      <td>0.001439</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2795919 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         predict      p_cn      p_cr      p_tn      p_tr  uplift_score\n",
       "0              0  0.462914  0.039050  0.459005  0.039032      0.001154\n",
       "1              2  0.459171  0.040068  0.460331  0.040430      0.000960\n",
       "2              0  0.461210  0.039238  0.459893  0.039659      0.001968\n",
       "3              2  0.451050  0.043022  0.451416  0.054511      0.041337\n",
       "4              0  0.462915  0.039050  0.459006  0.039029      0.001144\n",
       "...          ...       ...       ...       ...       ...           ...\n",
       "2795914        2  0.262453  0.210295  0.291141  0.236111      0.005958\n",
       "2795915        0  0.490065  0.081051  0.337942  0.090942      0.140252\n",
       "2795916        0  0.463028  0.039023  0.458692  0.039257      0.002219\n",
       "2795917        2  0.448384  0.051104  0.450981  0.049531     -0.006707\n",
       "2795918        0  0.462878  0.039047  0.458969  0.039106      0.001439\n",
       "\n",
       "[2795919 rows x 6 columns]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "uplift_predict_lgwum = h2o_gbm_lgwum.predict(h2o_test_df)\n",
    "\n",
    "result = uplift_predict_lgwum.as_data_frame()\n",
    "result.columns = ['predict', 'p_cn', 'p_cr', 'p_tn', 'p_tr']\n",
    "result['uplift_score'] = result.eval('\\\n",
    "                                p_cn/(p_cn + p_cr) \\\n",
    "                                + p_tr/(p_tn + p_tr) \\\n",
    "                                - p_tn/(p_tn + p_tr) \\\n",
    "                                - p_cr/(p_cn + p_cr)')\n",
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmwAAAF+CAYAAAA/Y0YoAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAio0lEQVR4nO3debxdZX3v8c+XIDNRJlslYgLFWqoWbRyqXEScUBS8iIjWisgV2+s80GKdEG3r3NZKVawD2KuIlNJYolgHwFIHwiiDSATUIBUEZKzI8Lt/7HWSzeEMKzlZ+6xz8nm/Xvt11vDstX/7ITl8s9Z61pOqQpIkSf210WwXIEmSpKkZ2CRJknrOwCZJktRzBjZJkqSeM7BJkiT1nIFNkiSp5+ZkYEvy6STXJrmoZfuDklyS5OIkn++6PkmSpPUpc/E5bEn2BG4Fjq+qR0zTdlfgRGDvqroxyQOr6tpR1ClJkrQ+zMkzbFV1JnDD8LYkuyT5apJzknw7ycObXa8AjqmqG5v3GtYkSdKcMicD2ySOBV5TVX8IvBn4x2b7w4CHJTkryXeT7DNrFUqSJK2DjWe7gPUhyVbAE4EvJRnbvGnzc2NgV2AvYBFwZpJHVtWvRlymJEnSOpkXgY3BmcJfVdXuE+xbBXyvqu4ErkzyIwYB7uwR1idJkrTO5sUl0aq6mUEYewFABv6g2X0Kg7NrJNmewSXSK2ahTEmSpHUyJwNbki8A3wF+N8mqJIcBfwwcluQC4GJg/6b5acD1SS4BvgUcUVXXz0bdkiRJ62JOPtZDkiRpQzInz7BJkiRtSAxskiRJPTfnRoluv/32tXjx4tkuQ5IkaVrnnHPOL6tqh5keZ84FtsWLF7NixYrZLkOSJGlaSX6yPo7jJVFJkqSeM7BJkiT1nIFNkiSp5wxskiRJPWdgkyRJ6jkDmyRJUs8Z2CRJknrOwCZJktRzBjZJkqSeM7BJkiT1nIFNkiSp5wxskiRJPWdgkyRJ6rmNZ7sASe0tOfLUKfdf+d59R1SJJGmUOjvDluTTSa5NctEk+5PkI0lWJrkwyWO6qkWSJGku6/KS6GeBfabY/yxg1+Z1OPCxDmuRJEmaszoLbFV1JnDDFE32B46vge8CD0jyoK7qkSRJmqtmc9DBjsDPhtZXNdvuI8nhSVYkWXHdddeNpDhJkqS+mBOjRKvq2KpaWlVLd9hhh9kuR5IkaaRmM7BdDTxkaH1Rs02SJElDZjOwLQNe2owWfQJwU1VdM4v1SJIk9VJnz2FL8gVgL2D7JKuAdwL3A6iqjwPLgWcDK4HbgUO7qkWSJGku6yywVdWLptlfwKu6+nxJkqT5Yk4MOpAkSdqQGdgkSZJ6zsAmSZLUcwY2SZKknjOwSZIk9ZyBTZIkqecMbJIkST1nYJMkSeo5A5skSVLPGdgkSZJ6zsAmSZLUcwY2SZKknjOwSZIk9ZyBTZIkqecMbJIkST1nYJMkSeo5A5skSVLPGdgkSZJ6zsAmSZLUcwY2SZKknjOwSZIk9ZyBTZIkqecMbJIkST1nYJMkSeo5A5skSVLPGdgkSZJ6zsAmSZLUcwY2SZKknjOwSZIk9ZyBTZIkqecMbJIkST1nYJMkSeo5A5skSVLPGdgkSZJ6zsAmSZLUcwY2SZKknjOwSZIk9dy0gS3Jpm22SZIkqRttzrB9p+U2SZIkdWDjyXYk+W1gR2DzJI8G0uxaCGwxgtokSZLEFIENeCbwMmAR8OGh7TcDf9lhTZIkSRoyaWCrquOA45I8v6r+ZYQ1SZIkaUibe9jOSvKpJF8BSLJbksM6rkuSJEmNNoHtM8BpwIOb9R8Br++qIEmSJN1bm8C2fVWdCNwDUFV3AXd3WpUkSZJWaxPYbkuyHVAASZ4A3NRpVZIkSVptqlGiY94ILAN2SXIWsANwYKdVSZIkabVpA1tVnZvkycDvMngW22VVdWfnlUmSJAloNzXVC4DNq+pi4HnAF5M8puvCJEmSNNDmHra3V9UtSfYAngp8CvhYt2VJkiRpTJvANjYidF/gk1V1KrBJdyVJkiRpWJvAdnWSTwAvBJYn2bTl+yRJkrQetAleBzF4cO4zq+pXwLbAEW0OnmSfJJclWZnkyAn275TkW0nOS3JhkmevTfGSJEkbgjajRG8HTh5avwa4Zrr3JVkAHAM8HVgFnJ1kWVVdMtTsbcCJVfWxJLsBy4HFa/UNJEmS5rkuL20+DlhZVVdU1W+AE4D9x7UpYGGzfH/g5x3WI0mSNCe1eXDuutoR+NnQ+irg8ePaHAV8LclrgC2Bp3VYjyRJ0pw024MHXgR8tqoWAc8GPpfkPjUlOTzJiiQrrrvuupEXKUmSNJvaPDj3gCSXJ7kpyc1Jbklyc4tjXw08ZGh9UbNt2GHAiQBV9R1gM2D78QeqqmOramlVLd1hhx1afLQkSdL80eYM2/uB/arq/lW1sKq2rqqF074LzgZ2TbIkySbAwQzmJB32UwYP4yXJ7zEIbJ5CkyRJGtImsP2iqi5d2wNX1V3Aqxk8EuRSBqNBL05ydJL9mmZvAl6R5ALgC8DLqqrW9rMkSZLmszaDDlYk+SJwCnDH2MaqOnnSd6xps5zBozqGt71jaPkS4Elti5UkSdoQtQlsC4HbgWcMbSuGns0mSZKk7rR5cO6hoyhEkiRJE2szSnRRkn9Ncm3z+pcki0ZRnCRJktoNOvgMg9GdD25eX262SZIkaQTaBLYdquozVXVX8/os4MPQJEmSRqRNYLs+yUuSLGheLwGu77owSZIkDbQJbC8HDgL+G7gGOBBwIIIkSdKItBkl+hNgv+naSZIkqRuTBrYkf15V70/yDwyeu3YvVfXaTiuTJEkSMPUZtrHpqFaMohBJkiRNbNLAVlVfbhZvr6ovDe9L8oJOq5IkSdJqbQYdvKXlNkmSJHVgqnvYngU8G9gxyUeGdi0E7uq6MEmSJA1MdQ/bzxncv7YfcM7Q9luAN3RZlCRJktaY6h62C4ALkny+qu4cYU2SJEkaMu1z2IDFSf4G2A3YbGxjVe3cWVWSJElare3k7x9jcN/aU4DjgX/usihJkiSt0SawbV5V3wBSVT+pqqOAfbstS5IkSWPaXBK9I8lGwOVJXg1cDWzVbVmSJEka0+YM2+uALYDXAn8I/AlwSJdFSZIkaY02k7+f3SzeChzabTmSJEkab6oH536ZCSZ9H1NV+3VSkSRJku5lqjNsH2x+HgD8NmtGhr4I+EWXRUmSJGmNqR6cewZAkg9V1dKhXV9OsqLzyiRJkgS0G3SwZZLVD8lNsgTYsruSJEmSNKzNYz3eAJye5AogwEOBV3ZalSRJklZrM0r0q0l2BR7ebPphVd3RbVmSJEkaM9Uo0b2r6ptJDhi3a5ckVNXJHdcmSZIkpj7D9mTgm8BzJ9hXgIFNkiRpBKYaJfrO5qcPy5UkSZpFU10SfeNUb6yqD6//ciRJkjTeVJdEtx5ZFZIkSZrUVJdE3zXKQiRJkjSxaR/rkWQz4DDg94HNxrZX1cs7rEuSJEmNNjMdfI7BXKLPBM4AFgG3dFmUJEmS1mgT2H6nqt4O3FZVxwH7Ao/vtixJkiSNaRPY7mx+/irJI4D7Aw/sriRJkiQNazOX6LFJtgHeDiwDtmqWJUmSNAJtAttnqupuBvev7dxxPZIkSRqnzSXRK5Mcm+SpSdJ5RZIkSbqXNoHt4cDXgVcBVyX5aJI9ui1LkiRJY6YNbFV1e1WdWFUHALsDCxlcHpUkSdIItDnDRpInJ/lH4BwGD889qNOqJEmStFqbmQ6uAs4DTgSOqKrbui5KkiRJa7QZJfqoqrq580okSZI0oTb3sBnWJEmSZlGre9gkSZI0ewxskiRJPddm0MGmwPOBxcPtq+ro7sqSJEnSmDaDDv4NuInBIz3u6LYcSZIkjdcmsC2qqn06r0SSJEkTanMP238leWTnlUiSJGlCbQLbHsA5SS5LcmGSHyS5sM3Bk+zTvG9lkiMnaXNQkkuSXJzk82tTvCRJ0oagzSXRZ63LgZMsAI4Bng6sAs5OsqyqLhlqsyvwFuBJVXVjkgeuy2dJkiTNZ5MGtiQLm4fm3rKOx34csLKqrmiOdwKwP3DJUJtXAMdU1Y0AVXXtOn6WJEnSvDXVGbbPA89hMDq0gAztK2DnaY69I/CzofVVwOPHtXkYQJKzgAXAUVX11enLliRJ2nBMGtiq6jnNzyUdf/6uwF7AIuDMJI+sql8NN0pyOHA4wE477dRhOZIkSf3T5UwHVwMPGVpf1GwbtgpYVlV3VtWVwI8YBLh7qapjq2ppVS3dYYcdOitYkiSpj7oMbGcDuyZZkmQT4GBg2bg2pzA4u0aS7RlcIr2iw5okSZLmnM4CW1XdBbwaOA24FDixqi5OcnSS/ZpmpwHXJ7kE+BZwRFVd31VNkiRJc1GbuUR3AVZV1R1J9gIeBRw//j6ziVTVcmD5uG3vGFou4I3NS5IkSRNoc4btX4C7k/wOcCyD+9J8wK0kSdKItAls9zSXN/838A9VdQTwoG7LkiRJ0pg2ge3OJC8CDgH+vdl2v+5KkiRJ0rA2ge1Q4I+Av6qqK5MsAT7XbVmSJEkaM+2gg2buz9cOrV8JvK/LoiRJkrRGm1GiTwKOAh7atA+DAZ7TTU0lSZKk9WDawAZ8CngDgzlF7+62HEmSJI3XJrDdVFVf6bwSSZIkTahNYPtWkg8AJwN3jG2sqnM7q0qSJEmrtQlsj29+Lh3aVsDe678cSZIkjddmlOhTRlGIJEmSJjbtc9iS3D/Jh5OsaF4fSnL/URQnSZKkdg/O/TRwC3BQ87oZ+EyXRUmSJGmNNvew7VJVzx9af1eS8zuqR5IkSeO0OcP2P0n2GFtpHqT7P92VJEmSpGFtzrD9GXBcc99agBuAl3VZlCRJktZoM0r0fOAPkixs1m/uuihJkiStMWlgS/KSqvrnJG8ctx2Aqvpwx7VJkiSJqc+wbdn83HqCfdVBLZIkSZrApIGtqj7RLH69qs4a3tcMPJAkSdIItBkl+g8tt0mSJKkDU93D9kfAE4Edxt3HthBY0HVhkiRJGpjqHrZNgK2aNsP3sd0MHNhlUZIkSVpjqnvYzgDOSPLZqvrJCGuSJEnSkDYPzr09yQeA3wc2G9tYVXt3VpUkSZJWazPo4P8BPwSWAO8CrgLO7rAmSZIkDWkT2Larqk8Bd1bVGVX1csCza5IkSSPS5pLonc3Pa5LsC/wc2La7kiRJkjSsTWB7TzPx+5sYPH9tIfCGTquSNkBLjjx1tkuQJPVUm8D2vaq6CbgJeErH9UiSJGmcNvewnZXka0kOS7JN5xVJkiTpXqYNbFX1MOBtDB7rcU6Sf0/yks4rkyRJEtDuDBtV9f2qeiPwOOAG4LhOq5IkSdJq0wa2JAuTHJLkK8B/AdcwCG6SJEkagTaDDi4ATgGOrqrvdFuOJEmSxpsysCVZAJxcVW8aUT2SJEkaZ8pLolV1N/DEEdUiSZKkCbS5JHp+kmXAl4DbxjZW1cmdVSVJkqTV2gS2zYDruff8oQUY2CRJkkZg2sBWVYeOohBJkiRNrM1jPR6W5BtJLmrWH5Xkbd2XJkmSJGj34NxPAm8B7gSoqguBg7ssSpIkSWu0CWxbVNX3x227q4tiJEmSdF9tAtsvk+zCYKABSQ5kMNuBJEmSRqDNKNFXAccCD09yNXAl4OTvkiRJI9JmlOgVwNOSbAlsVFW3dF+WJEmSxrQZJfq6JAuB24G/TXJukmd0X5okSZKg3T1sL6+qm4FnANsBfwK8t9OqJEmStFqbwJbm57OB46vq4qFtkiRJ6libwHZOkq8xCGynJdkauKfbsiRJkjSmzSjRw4DdgSuq6vYk2wFOVyVJkjQibUaJ3pNkMfCSJAX8Z1X9a+eVSZIkCWg3SvQfgT8FfgBcBLwyyTFdFyZJkqSBNvew7Q08s6o+U1WfYXAv21PbHDzJPkkuS7IyyZFTtHt+kkqytF3ZkiRJG442gW0lsNPQ+kOAy6d7U5IFwDHAs4DdgBcl2W2CdlsDrwO+16ZgSZKkDc2kgS3Jl5MsA7YGLk1yepLTgUubbdN5HLCyqq6oqt8AJwD7T9Du3cD7gF+vbfGSJEkbgqkGHXxwhsfeEfjZ0Poq4PHDDZI8BnhIVZ2a5IjJDpTkcOBwgJ122mmyZpIkSfPSpIGtqs4YW07yW8Bjm9XvV9W1M/3gJBsBHwZeNl3bqjqWwQT0LF26tGb62ZIkSXNJm1GiBwHfB14AHAR8L8mBLY59NYP73cYsaraN2Rp4BHB6kquAJwDLHHggSZJ0b20enPtW4LFjZ9WS7AB8HThpmvedDeyaZAmDoHYw8OKxnVV1E7D92Hpzf9ybq2rF2nwBSZKk+a7NKNGNxl0Cvb7N+6rqLuDVwGkMBiqcWFUXJzk6yX7rVK0kSdIGqM0Ztq8mOQ34QrP+QmB5m4NX1fLxbavqHZO03avNMSVJkjY0baamOiLJAcAezaZjnZpKkiRpdNqcYaOqTgZO7rgWSZIkTaDNPWySJEmaRQY2SZKknptqaqpvND/fN7pyJEmSNN5U97A9KMkTgf2SnABkeGdVndtpZZIkSQKmDmzvAN7OYIaCD4/bV8DeXRUlSZKkNaaaS/Qk4KQkb6+qd4+wJkmSJA1p8xy2dzczE+zZbDq9qv6927IkSZI0ps3k738DvA64pHm9Lslfd12YJEmSBto8OHdfYPequgcgyXHAecBfdlmYJEmSBto+h+0BQ8v376AOSZIkTaLNGba/Ac5L8i0Gj/bYEziy06okSZK0WptBB19Icjrw2GbTX1TVf3dalSRJklZrO/n7NcCyjmuRJEnSBJxLVJIkqecMbJIkST03ZWBLsiDJD0dVjCRJku5rysBWVXcDlyXZaUT1SJIkaZw2gw62AS5O8n3gtrGNVbVfZ1VJkiRptTaB7e2dVyFJkqRJtXkO2xlJHgrsWlVfT7IFsKD70iRJkgTtJn9/BXAS8Ilm047AKR3WJEmSpCFtHuvxKuBJwM0AVXU58MAui5IkSdIabQLbHVX1m7GVJBsD1V1JkiRJGtYmsJ2R5C+BzZM8HfgS8OVuy5IkSdKYNoHtSOA64AfAK4HlwNu6LEqSJElrtBklek+S44DvMbgUellVeUlUkiRpRKYNbEn2BT4O/BgIsCTJK6vqK10XJ0mSpHYPzv0Q8JSqWgmQZBfgVMDAJkmSNAJt7mG7ZSysNa4AbumoHkmSJI0z6Rm2JAc0iyuSLAdOZHAP2wuAs0dQmyRJkpj6kuhzh5Z/ATy5Wb4O2LyziiRJknQvkwa2qjp0lIVIkiRpYm1GiS4BXgMsHm5fVft1V5YkSZLGtBklegrwKQazG9zTaTWSJEm6jzaB7ddV9ZHOK5EkSdKE2gS2v0/yTuBrwB1jG6vq3M6qkiRJ0mptAtsjgT8B9mbNJdFq1iVJktSxNoHtBcDOVfWbrouRJEnSfbUJbBcBDwCu7bYUSTO15MhTp21z5Xv3HUElkqT1qU1gewDwwyRnc+972HyshyRJ0gi0CWzv7LwKSZIkTWrawFZVZ4yiEEmSJE2szUwHtzAYFQqwCXA/4LaqWthlYZIkSRpoc4Zt67HlJAH2B57QZVGSJElaY6O1aVwDpwDP7KYcSZIkjdfmkugBQ6sbAUuBX3dWkSRJku6lzSjR5w4t3wVcxeCyqCRJkkagzT1sh46iEEmSJE1s0sCW5B1TvK+q6t0d1CNJkqRxpjrDdtsE27YEDgO2AwxskiRJIzDpKNGq+tDYCzgW2Bw4FDgB2LnNwZPsk+SyJCuTHDnB/jcmuSTJhUm+keSh6/g9JEmS5q0pH+uRZNsk7wEuZHA27jFV9RdVNe1E8EkWAMcAzwJ2A16UZLdxzc4DllbVo4CTgPevw3eQJEma1yYNbEk+AJwN3AI8sqqOqqob1+LYjwNWVtUVVfUbBmfm7jW6tKq+VVW3N6vfBRatVfWSJEkbgKnOsL0JeDDwNuDnSW5uXrckubnFsXcEfja0vqrZNpnDgK9MtCPJ4UlWJFlx3XXXtfhoSZKk+WPSQQdVtVazIMxEkpcweCDvkyep5VgG99GxdOnSmqiNJEnSfNXmwbnr6mrgIUPri5pt95LkacBbgSdX1R0d1iNJkjQndXkW7Wxg1yRLkmwCHAwsG26Q5NHAJ4D92gxkkCRJ2hB1Ftiq6i7g1cBpwKXAiVV1cZKjk+zXNPsAsBXwpSTnJ1k2yeEkSZI2WF1eEqWqlgPLx217x9Dy07r8fEmSpPlgZAMLJEmStG4MbJIkST1nYJMkSeo5A5skSVLPGdgkSZJ6zsAmSZLUcwY2SZKknjOwSZIk9ZyBTZIkqec6nelAUv8sOfLUKfdf+d59R1SJJKktz7BJkiT1nIFNkiSp5wxskiRJPWdgkyRJ6jkDmyRJUs8Z2CRJknrOwCZJktRzBjZJkqSeM7BJkiT1nIFNkiSp5wxskiRJPWdgkyRJ6jkDmyRJUs8Z2CRJknrOwCZJktRzBjZJkqSeM7BJkiT1nIFNkiSp5wxskiRJPWdgkyRJ6jkDmyRJUs8Z2CRJknrOwCZJktRzG892AZL6ZcmRp07b5sr37juCSiRJYzzDJkmS1HMGNkmSpJ4zsEmSJPWcgU2SJKnnDGySJEk9Z2CTJEnqOQObJElSz/kcNklrbbpntfmcNklavzzDJkmS1HMGNkmSpJ7zkqg0Im2mfJIkaSIGNknrnfe4SdL6ZWCTNHJOMC9Ja8d72CRJknrOM2zSeuD9aeufl1UlaQ0Dm6Q5ycuqkjYkBjapBc+gSZJmk4FNGzzD2Py1Pv7bepZOUh90GtiS7AP8PbAA+Keqeu+4/ZsCxwN/CFwPvLCqruqyJs0vhi11zdAnqQ86C2xJFgDHAE8HVgFnJ1lWVZcMNTsMuLGqfifJwcD7gBd2VZNGyzAlDcyVvwsGS6m/ujzD9jhgZVVdAZDkBGB/YDiw7Q8c1SyfBHw0SaqqOqxrVs2VX9ySNjz+fho9Q7La6jKw7Qj8bGh9FfD4ydpU1V1JbgK2A3452UF/cPVN/lKRJM0L/v9Mbc2JQQdJDgcOb1Zvvep9z7lsNuuZwPZMETJ1L/ZVO/ZTO/ZTe/ZVO/ZTe/ZVO7+7Pg7SZWC7GnjI0PqiZttEbVYl2Ri4P4PBB/dSVccCx3ZU54wlWVFVS2e7jrnAvmrHfmrHfmrPvmrHfmrPvmonyYr1cZwup6Y6G9g1yZIkmwAHA8vGtVkGHNIsHwh8cz7fvyZJkrQuOjvD1tyT9mrgNAaP9fh0VV2c5GhgRVUtAz4FfC7JSuAGBqFOkiRJQzq9h62qlgPLx217x9Dyr4EXdFnDiPT2cm0P2Vft2E/t2E/t2Vft2E/t2VftrJd+ilcgJUmS+q3Le9gkSZK0HhjYWkqybZL/SHJ583ObSdod0rS5PMkhE+xfluSi7iuePTPpqyRbJDk1yQ+TXJzkvRO9dy5Lsk+Sy5KsTHLkBPs3TfLFZv/3kiwe2veWZvtlSZ450sJHbF37KcnTk5yT5AfNz71HXvwIzeTPU7N/pyS3JnnzyIqeJTP8u/eoJN9pfi/9IMlmIy1+hGbwd+9+SY5r+ufSJG8ZefEj1qKv9kxybpK7khw4bt+UeeE+qspXixfwfuDIZvlI4H0TtNkWuKL5uU2zvM3Q/gOAzwMXzfb36WtfAVsAT2nabAJ8G3jWbH+n9dg3C4AfAzs33+8CYLdxbf4v8PFm+WDgi83ybk37TYElzXEWzPZ36mE/PRp4cLP8CODq2f4+feynof0nAV8C3jzb36evfcXgfu8LgT9o1rfz796E/fRi4IRmeQvgKmDxbH+nWe6rxcCjGMybfuDQ9inzwkQvz7C1tz9wXLN8HPC8Cdo8E/iPqrqhqm4E/gPYByDJVsAbgfd0X+qsW+e+qqrbq+pbAFX1G+BcBs/wmy9WT9nWfL+xKduGDfffScBTk6TZfkJV3VFVVwIrm+PNR+vcT1V1XlX9vNl+MbB5kk1HUvXozeTPE0meB1zJoJ/mu5n01TOAC6vqAoCqur6q7h5R3aM2k34qYMsMnqu6OfAb4ObRlD0rpu2rqrqqqi4E7hn33knzwmQMbO39VlVd0yz/N/BbE7SZaDquHZvldwMfAm7vrML+mGlfAZDkAcBzgW90UONsmfZ7M27KNmBsyrY2750vZtJPw54PnFtVd3RU52xb535q/hH5F8C7RlBnH8zkz9TDgEpyWnN5689HUO9smUk/nQTcBlwD/BT4YFXd0HXBs2gmv5PX+r1zYmqqUUnydeC3J9j11uGVqqokrYfXJtkd2KWq3jD+/pG5qqu+Gjr+xsAXgI9U1RXrVqU2ZEl+H3gfg7Mjuq+jgL+tqlubE26a3MbAHsBjGfyj+xtJzqmq+fSPyfXhccDdwIMZXOb7dpKv+zt8/TCwDamqp022L8kvkjyoqq5J8iDg2gmaXQ3sNbS+CDgd+CNgaZKrGPT5A5OcXlV7MUd12FdjjgUur6q/m3m1vTKTKdvavHe+mNHUdkkWAf8KvLSqftx9ubNmJv30eODAJO8HHgDck+TXVfXRzqueHTPpq1XAmVX1S4Aky4HHML/O/o+ZST+9GPhqVd0JXJvkLGApg/uz5qOZ/E6e7v+B9+El0faGp9E6BPi3CdqcBjwjyTYZjIx8BnBaVX2sqh5cVYsZ/CvtR3M5rLWwzn0FkOQ9DH4BvL77UkduJlO2LQMObkZoLQF2Bb4/orpHbZ37qbmUfiqDgS9njargWbLO/VRV/6uqFje/l/4O+Ot5HNZgZn/3TgMemcEo9o2BJwOXjKjuUZtJP/0U2BsgyZbAE4AfjqTq2dGmryYz6f8DJzXboyzmyovB9flvAJcDXwe2bbYvBf5pqN3LGdwMvhI4dILjLGb+jxJd575i8K+MAi4Fzm9e/2e2v9N67p9nAz9iMLrorc22o4H9muXNGIzaW8kgkO089N63Nu+7jHk0enZ99hPwNgb30Zw/9HrgbH+fvvXTuGMcxTwfJTrTvgJewmBwxkXA+2f7u/Sxn4Ctmu0XMwi0R8z2d+lBXz2WwRna2xichbx46L1T5oXxL2c6kCRJ6jkviUqSJPWcgU2SJKnnDGySJEk9Z2CTJEnqOQObJElSzxnYJPVSksVJLhq37agkb57mfS9L8tFm+U+TvLRZfniS85Ocl2SXJC/urnpJWr8MbJLmrar6eFUd36w+Dzipqh7N4OnkIwlszYNWJWlG/EUiaU5KcjpwAYOnzm8MvLyqvj+uzVHArQwe4vl64O4kTwU2B34vyfnAcVX1t0PveRDwRWBhc9w/q6pvJ9kH+GtgAfDLqnpqkm2BTwM7M5hj8vCqurD53F2a7T9N8lrg48BOzce8vub/LAyS1iMDm6S5bIuq2j3JngyC0yMmalRVy5N8HLi1qj6YZC8GT/Z/zgTNX8xgSrm/SrIA2CLJDsAngT2r6somqAG8Czivqp6XZG/geGD3Zt9uwB5V9T9JPs9govX/TLITgylofm89fH9JGwgDm6S+mmwaluHtXwCoqjOTLGzmEZ2ps4FPJ7kfcEpVnd8EvDOr6srm825o2u4BPL/Z9s0k2yVZ2OxbVlX/0yw/DdgtydhnLEyyVVXduh7qlbQBMLBJ6qvrgW3GbdsWuHJofXyom/Fce0342xPYF/hskg8DN67DoW4bWt4IeEJV/Xqm9UnaMDnoQFIvNWefrmkuNdJchtwH+M+hZi9s9u0B3FRVN7U8/C3A1hPtSPJQ4BdV9Ungn4DHAN8F9kyyZKgWgG8Df9xs24vBvW03T3DYrwGvGfqM3VvWKUmAZ9gk9dtLgWOas1wA76qqHw/t/3WS84D7AS9fi+NeyGAAwgXAZ4cHHQB7AUckuZPBgIWXVtV1SQ4HTk6yEXAt8HTgKAaXTy9kMOjgkEk+77XN97iQwe/dM4E/XYt6JW3gUjXjKwiSNHLNKNE3V9WK2a5FkrrmJVFJkqSe8wybJElSz3mGTZIkqecMbJIkST1nYJMkSeo5A5skSVLPGdgkSZJ6zsAmSZLUc/8fXt90iyjarwMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_uplift_score(result.uplift_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parse progress: |████████████████████████████████████████████████████████████████| (done) 100%\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style='margin: 1em 0 1em 0;'>ModelMetricsBinomialUplift: \n",
       "** Reported on test data. **\n",
       "\n",
       "AUUC: 21114.069636270284\n",
       "AUUC normalized: 0.7321073609877702</pre>\n",
       "<div style='margin: 1em 0 1em 0;'>\n",
       "<style>\n",
       "\n",
       "#h2o-table-13.h2o-container {\n",
       "  overflow-x: auto;\n",
       "}\n",
       "#h2o-table-13 .h2o-table {\n",
       "  /* width: 100%; */\n",
       "  margin-top: 1em;\n",
       "  margin-bottom: 1em;\n",
       "}\n",
       "#h2o-table-13 .h2o-table caption {\n",
       "  white-space: nowrap;\n",
       "  caption-side: top;\n",
       "  text-align: left;\n",
       "  /* margin-left: 1em; */\n",
       "  margin: 0;\n",
       "  font-size: larger;\n",
       "}\n",
       "#h2o-table-13 .h2o-table thead {\n",
       "  white-space: nowrap; \n",
       "  position: sticky;\n",
       "  top: 0;\n",
       "  box-shadow: 0 -1px inset;\n",
       "}\n",
       "#h2o-table-13 .h2o-table tbody {\n",
       "  overflow: auto;\n",
       "}\n",
       "#h2o-table-13 .h2o-table th,\n",
       "#h2o-table-13 .h2o-table td {\n",
       "  text-align: right;\n",
       "  /* border: 1px solid; */\n",
       "}\n",
       "#h2o-table-13 .h2o-table tr:nth-child(even) {\n",
       "  /* background: #F5F5F5 */\n",
       "}\n",
       "\n",
       "</style>      \n",
       "<div id=\"h2o-table-13\" class=\"h2o-container\">\n",
       "  <table class=\"h2o-table\">\n",
       "    <caption>AUUC table (number of bins: 72): All types of AUUC value</caption>\n",
       "    <thead><tr><th>uplift_type</th>\n",
       "<th>qini</th>\n",
       "<th>lift</th>\n",
       "<th>gain</th></tr></thead>\n",
       "    <tbody><tr><td>AUUC value</td>\n",
       "<td>17984.8660027</td>\n",
       "<td>0.0243477</td>\n",
       "<td>21114.0696363</td></tr>\n",
       "<tr><td>AUUC normalized</td>\n",
       "<td>0.7336534</td>\n",
       "<td>0.0243477</td>\n",
       "<td>0.7321074</td></tr>\n",
       "<tr><td>AUUC random value</td>\n",
       "<td>12495.7909868</td>\n",
       "<td>0.0052580</td>\n",
       "<td>14700.9253147</td></tr></tbody>\n",
       "  </table>\n",
       "</div>\n",
       "</div>\n",
       "<pre style='margin: 1em 0 1em 0;'>Qini value: 5489.075015965091</pre>\n",
       "<div style='margin: 1em 0 1em 0;'>\n",
       "<style>\n",
       "\n",
       "#h2o-table-14.h2o-container {\n",
       "  overflow-x: auto;\n",
       "}\n",
       "#h2o-table-14 .h2o-table {\n",
       "  /* width: 100%; */\n",
       "  margin-top: 1em;\n",
       "  margin-bottom: 1em;\n",
       "}\n",
       "#h2o-table-14 .h2o-table caption {\n",
       "  white-space: nowrap;\n",
       "  caption-side: top;\n",
       "  text-align: left;\n",
       "  /* margin-left: 1em; */\n",
       "  margin: 0;\n",
       "  font-size: larger;\n",
       "}\n",
       "#h2o-table-14 .h2o-table thead {\n",
       "  white-space: nowrap; \n",
       "  position: sticky;\n",
       "  top: 0;\n",
       "  box-shadow: 0 -1px inset;\n",
       "}\n",
       "#h2o-table-14 .h2o-table tbody {\n",
       "  overflow: auto;\n",
       "}\n",
       "#h2o-table-14 .h2o-table th,\n",
       "#h2o-table-14 .h2o-table td {\n",
       "  text-align: right;\n",
       "  /* border: 1px solid; */\n",
       "}\n",
       "#h2o-table-14 .h2o-table tr:nth-child(even) {\n",
       "  /* background: #F5F5F5 */\n",
       "}\n",
       "\n",
       "</style>      \n",
       "<div id=\"h2o-table-14\" class=\"h2o-container\">\n",
       "  <table class=\"h2o-table\">\n",
       "    <caption>AECU values table: All types of AECU value</caption>\n",
       "    <thead><tr><th>uplift_type</th>\n",
       "<th>qini</th>\n",
       "<th>lift</th>\n",
       "<th>gain</th></tr></thead>\n",
       "    <tbody><tr><td>AECU value</td>\n",
       "<td>5489.0750160</td>\n",
       "<td>0.0190897</td>\n",
       "<td>6413.1443216</td></tr></tbody>\n",
       "  </table>\n",
       "</div>\n",
       "</div>"
      ],
      "text/plain": [
       "ModelMetricsBinomialUplift: \n",
       "** Reported on test data. **\n",
       "\n",
       "AUUC: 21114.069636270284\n",
       "AUUC normalized: 0.7321073609877702\n",
       "\n",
       "AUUC table (number of bins: 72): All types of AUUC value\n",
       "uplift_type        qini      lift        gain\n",
       "-----------------  --------  ----------  --------\n",
       "AUUC value         17984.9   0.0243477   21114.1\n",
       "AUUC normalized    0.733653  0.0243477   0.732107\n",
       "AUUC random value  12495.8   0.00525799  14700.9\n",
       "\n",
       "Qini value: 5489.075015965091\n",
       "\n",
       "AECU values table: All types of AECU value\n",
       "uplift_type    qini     lift       gain\n",
       "-------------  -------  ---------  -------\n",
       "AECU value     5489.08  0.0190897  6413.14"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lgwum_predict = h2o.H2OFrame(result['uplift_score'].tolist())\n",
    "perf_lgwum = h2o.make_metrics(lgwum_predict, h2o_test_df[response_column], treatment=h2o_test_df[treatment_column], auuc_type=\"gain\", auuc_nbins=81)\n",
    "perf_lgwum"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAABHh0lEQVR4nO3deZzN1f/A8dfb2Jes5ZulLOlbyL61kKKSivSTpGxJyVKSFm2UkspSpKLSrlTiS6VNhhRCirQhyYQsYxv7mPfvj/MZbmOWO+Pe+7kz9/18PO7j3ns+2/vcy33P53zO5xxRVYwxxphok8/vAIwxxpj0WIIyxhgTlSxBGWOMiUqWoIwxxkQlS1DGGGOikiUoY4wxUckSlIlZIjJMRN7yO45giEgPEVkQ8D5JRKp5r4uIyCwR2SUi7/sXZXQQkdki0j3U65rIswRlIk5EuojIUu9HdpP3I3GB33FlRkReE5HHTmD7eBG5OU1ZSxFJyMn+VLW4qv7hve0IlAfKquq1wcQqzu0i8pOI7BWRBBF5X0TOyUk80URVL1fV10O9rok8S1AmokRkEPAMMAL3o3oa8DzQ3sewcrvTgd9VNTkb2zwL3AHcDpQBzgRmAFdk9+Aikj+72xgTFFW1hz0i8gBKAknAtZms8xrwWMD7lkBCwPs/gbuBFcBe4BVcopsN7AG+BEqnt23A9q2918OAtwKWvQ9sBnYB84FaXvktwGHgkBf/LK+8AjAN2AqsA27PpF7xwM1pytLWTXEJ4w9gG/A0kM9b1gNYkGbdM4BHvLgOe7Hdml6saY5bAzgCNAk23gyO3w9Y7dX9BWBUmn38DxiU3c8qiH9HccAo7zP6w4tDgfxpY0+N21t/h3fsyzP7XuwRPQ87gzKRdC5QGJh+gvv5P+AS3F/9V+GS0/3AybhWgdtzuN/ZuB/vU4DvgbcBVHWS9/opdU1rV4lIPmAW8CNQEWgFDBSRy3JaKU8HoBHQAHdWeVNmK6vqUNzZ6FQvtolpY01ns1a4xPjdCcZ6NdAUqAm8A1wnIgIgIqWBS4F3w/BZ9QauBOrjPquOWazfFPgNKAc8BbySGqeJbpagTCSVBbZp9pqi0jNeVf9R1b+Br4HFqrpcVQ/gkl/9nOxUVSer6h5VPYg7u6orIiUzWL0xcLKqPqqqh9RdD3oJ6JyTYwd4UlUTVfUvXFPo9Se4v/SUBTaFYD9PeLHux30PCjT3lnUEFqrqRkL/WXUCnlHVDaqaCDyRxfrrVfUlVT0CvA6cijvrNlHO2o5NJG0HyolI/hNMUv8EvN6fzvvi2d2hiMQBjwPX4s7EUrxF5XBNfmmdDlQQkZ0BZXG4H+r0JAMF0pQVwDXHBdoQ8Ho9rmks1LbjfqRP1NFYVVVF5F1cQp0PdAFSe0gG/VmJyGnAzwH7Te+7rMDxn1NmNgfsb5938pTtfyMm8uwMykTSQuAgrmkoI3uBogHv/3MCx/vXvrwkdHIG63bBNam1xl0rq5K6mfecdtj/DcA6VS0V8Cihqm0z2P9fAftMVZXjf1wrB7w+DdiYwf4yk9UUBXOASiLSKJN1gvke0h7nHaCjiJyOa1ab5pUH/Vmp6l9e02TxDJITuLO/tJ+TyYMsQZmIUdVdwMPABBG5WkSKikgBEblcRJ7yVvsBaCsiZUTkP8DAEzjk70BhEblCRAoADwKFMli3BC55bsf9MI9Is/wfoFrA+++APSJyr3cfUpyI1BaRxhnsfyrQU0SaeF28zwTuBN5Ns97dIlJaRCrjetlNDbKumcX6L6q6Gtdz8h2vq3tBESksIp1F5D5vtR+Aa7zv6AygV1YHVdXluI4LLwOfqepOb1F2P6usvAfcLiKVvGtd92W1gcmdLEGZiFLV0cAgXLLYivvruj+uizPAm7iL6X8Cn5OzH+jUY+0C+uJ+MP/GnRVkdN/RG7izmb9xTUyL0ix/BagpIjtFZIZ3PeNKoB6uZ1jqD3O616xU9TPcD+mruCbDT3DXQyalWfV/wDJcgvjYO252/SvWDNa5HXgOmADsBNbiOmjM8paPxfUE/MeL8+0gjz0FdxY6JbUgu59VEF4CPsP9O/ke+DCH+zFRTlRtwkJjooGIKFBDVdf4HUtuIiJVcImvQAg64JgoYmdQxhhjopIlKGOMMVHJmviMMcZEJTuDMsYYE5UsQRljjIlKlqBMnuZNc7FDRAqlKctw6gsRqSIimnaU7rTTWIjIqSLyijdlyB4R+VVEHhGRYtmI7xQReUdENnrzOX0jIk3THGOmt1y9HmuB21cUkf+JSKI3ZUafNMvVm04jyXu8HLBsdkB5kogcEpGVGcTZTES+8I6z1Zua49SA5YVE5EUR+cdbZ5aIVAxY3l/cFCsHReS1NPuu6S3b4T2+FJGawX6GJu+yBGXyLO/HvDluxIN2Id53GdzIGEWAc1W1BG4A21JA9WzsqjiwBGiIm/bideBjEUkdRSEF+BQ3QG563sJ1sS6PmypjhIhclGadugGjMxxNzOrmQgocteFb3Iju6SmNu2erCm7ooj24e7pS3YEbDLgObiiiHcD4gOUbgceAyenseyNu7L4yuKGlZnL8DcwmBtlYfCYv64a74XYx0J2Mf3xzYhDuR/pGVU0BUNUNuB/qoHkDp44JKJokIqOA/wLLVPUf4Pm0Z3MAXhJrCXRS1cPAjyLyAW4E9LnZiSMgmffIIM7ZadZ/DpgXUFQVN3rEP97yqYH1UtUPvfJGQKU0+96Ju1kYERHcVCBnZCd+kzfZGZTJy7rhRkB4G7hMREI5gnVr4MPU5JQeEVnhjeaQ3uP5DLapBxQEgrlZV9I8p76unWa9+SKyWUQ+TNtEGKAb8LWq/hnEcQFaAKsC3r8CnC8iFUSkKHADbvqSoIkbTPYA7swr7VBTJgZZgjJ5krgp5E8H3lPVZbihfLqE8BBZTlmhqnXSDJAa+OibTswn4YZ6esQbpilTqroH+AZ4yBtLrwGuKTBwkNcLcc1yZ+Ga0j5K72wMl6Bey+qYXpx1cGMq3h1QvBo3bNXfwG7gbODRYPaXSlVL4YY/6g8sz862Jm+yBGXyqu7A56q6zXs/xSuDrKe+SA4oy2idUE1ZAYCIFMGNg7dIVbOa3yjQDbjmtQ24WW3fImC8QVWd783BtBPX/FgVlzwCj30BbrTyD4KI8wzcmdEdqho4XcYE3EC8ZYFiuPHxsnUG5cW7F3gReENETsnu9iZvsQRl8hzvx74TcKHXtLUZN3J4XRGpS9ZTX2zCJaLM1vkS6CButtiM4liVppdc4OPFgPUK4QbLTcBN2R40VV2vqleq6smq2hTXySCzmXKVfzcJgkvcH6pqUmbHEjeNxpfAcFV9M83iesBr3gSGB3HNdE1EpFw2qpMqH+4ssGJWK5q8zRKUyYuuxl1or4n74ayHO2v4GteUlenUF97o29OAx0WkrLgpQa739pd6VjAGOAl43fvhTu3yPcZrAkNVawX2kkvz6ONtUwB35rIf6J7eNS0RKcyxaUIKee9Tl50tIiXETZlxI26a9THesloiUk/c9BbFgdG4JrhfArZPTeavZfaBel3GvwKeU9UX01llCdBNREp6deoLbEw9gxWR/F7ccUCc1ySZ31t2iYjU9+I8yYt/R2CcJkapqj3skaceuG7Zo9Mp74SbXTU/rqfbKtz1kjW4qTDyBaxbmmPTdOzAXes5P83+KuC6TW/G9ej7FRgKFM1GrBfizmr2AUkBj+YB62jaR8CygbhpS/YCC4BGAcsuBn7zlm3BnaXVSHP863FnhZJObKuAG7zXQ71jB8aYFLBuWVxnlC24HnkLgCYBy4elU49h3rJrvc8uyavLx0Adv/8d2cP/h43FZ4wxJipZE58xxpioZAnKGGNMVLIEZYwxJipZgjLGGBOVYm4svnLlymmVKlVyvP3evXspVizowapzLatn3hIr9YTYqWtequeyZcu2qerJactjLkFVqVKFpUuX5nj7+Ph4WrZsGbqAopTVM2+JlXpC7NQ1L9VTRNanV25NfMYYY6KSJShjjDFRKWwJSkQqi8hcEfnZG5PsDq98mIj8LSI/eI+2AdsMEZE1IvKbiFwWUN7GK1sjIvcFlFcVkcVe+VQRKRiu+hhjjImscJ5BJQN3qWpNoBnQL2Aa57GqWs97fAJu2megM1ALaIObpC1OROJwIyVfjhsL7fqA/Tzp7esM3HA0vcJYH2OMMREUtgSlqptU9Xvv9R7cwI+ZjU7cHnhXVQ+q6jrc+GhNvMcaVf1DVQ/hBvRs7828eTHHpgh4HTdIqDHGmDwgItegvFk86+Om3gbo7802OllESntlFXFz2qRK8MoyKi8L7FTV5DTlxhhj8oCwdzP3hvmfBgxU1d0i8gIwHDea8XDcFAA3hTmGW4BbAMqXL098fHyO95WUlHRC2+cWVs+8JVbqCbFTV7/rqeoe+cJ4mhPWBOXNCzMNeFtVPwRQ1X8Clr8EfOS9/RuoHLB5Ja+MDMq3A6VEJL93FhW4/r+o6iRgEkCjRo30RO4dyEv3HmTG6pm3xEo9IXbqGop6HjoEe/a4x+7d2Xv++msoVw4WLYLq1UNTp7TClqC8a0SvAL+o6piA8lNVdZP3tgPwk/d6JjBFRMbg5tmpgZsZVIAaIlIVl4A6A11UVUVkLtARd12qO/C/cNXHGGOiweHDLkls3lyYFSuyl1TSlh08GNwxixaFk06CEiWOPQNs27ad3r2L8uWXRcJyJhXOM6jzga7AShH5wSu7H9cLrx6uie9PvCmuVXWViLwH/IzrAdhP3cymiEh/4DPcbJyTVXWVt797gXdF5DFgOS4hGmNMVDl8OGdnKek9H0sqzTI9ZtGi/04oJ50ElSsfn2iyei5eHPKnyRRHjhwhLi6O8eNLMmTIF6xZczlnnhn6zy1sCUpVF+DOftL6JJNtHgceT6f8k/S2U9U/cL38jDEmpFKTSigSy4EDwR2zSJHjE0Tlyuknjo0bf6Vx47PSTSzpJZVQOHjwIM899xwvvvgiS5YsoX//UnTs2IZTTw39sSAGx+IzxuRdrvkrP+vXZz+JpC0LR1LJ7EylRInsJZX4+M20bHlWzj6obFJVZsyYwd13383atWtp06YNe/bsoVSpUpx6anrnIaFhCcoY46vk5NA1f7mkckGWxyxS5PgkUbEinH129pq/sptUcqO9e/dy5ZVXEh8fT82aNZk9ezZt2rSJyLHz+EdrjAmH1KQSisSyf39wxyxc+PgEUbEinHXWv8v/+WcNDRqckWliyetJJRT2799PkSJFKFasGKeffjrPP/88vXv3Jn8EPzz7moyJEdGUVII9Owl8XaBAcMeMj0+gZcszcv5Bxbj9+/czevRoxo4dy5IlS6hWrRqvvfaaL7FYgjImih05EpqEsnNn86C7FBcufHzCqFAh+72/spNUjP9UlXfeeYf77ruPDRs2cM011xAXF+drTJagjIkCqvDLLzBnjnusWgUbN8K+fcFtX6jQ8Qni1FPhzDPd+507N1KrVuWgEoslldiTnJzMRRddxIIFC6hfvz5vvvkmF154od9hWYIyxi8JCccS0pdfwibv9vXq1aFhQ2jXDkqWDK75q2AWE83Ex6+lZcvKma9kYs62bdsoV64c+fPnp3Xr1vTq1Ytu3bqRL5zjF2WDJShjImTnToiPd8lozhz49VdXfvLJ0KoVtG7tnqtU8TFIExP27NnDyJEjGTNmDJ9//jnNmzdn6NChfod1HEtQxoTJgQOwcKFLSF9+CUuXQkqKu8P/wguhd2+XlGrXDu+Am8akOnLkCK+99hoPPvggmzdv5oYbbqBKFP9FZAnKmBBJSYEffjiWkBYscL3d4uKgaVN48EGXkJo2zbpJzphQU1Vat25NfHw85557LjNmzKBp06Z+h5UpS1DG5JAqrF177BrSV19BYqJbVrs23HKLS0gtWrjrRcb4Yd26dZx++unky5ePbt260adPHzp16oQbzzu6WYIyJhv++cclotSktH69K69cGdq3d9eQLr6YsI1NZkywduzYwfDhw3nuued45ZVX6Nq1Kz179vQ7rGyxBGVMJpKSYP78Yx0bVqxw5aVKuUR0770uKdWoAbngD1ITAw4fPszEiRMZNmwYiYmJ9OrVi0suucTvsHLEEpQxAQ4fhu++O5aQFi50IzAUKgQXXABPPOESUoMG7tqSMdHmmmuu4aOPPuLiiy9mzJgx1K1b1++QcswSlIlpqvDTT8c6Nsyb586aRNy9SIMHu+tI553nBhg1JhqtWrWKqlWrUrRoUQYMGEDv3r256qqrcsV1psxYgjIx56+/jl1Dmj37PHbscOVnngldu7qE1LIllCnja5jGZGnr1q08/PDDTJo0ieHDh3P//fdz6aWX+h1WyFiCMnleYiLMnXssKa1e7crLl4eGDXfQpUt5WrWC007zN05jgnXw4EHeffdd3nnnHfbu3Uu/fv249dZb/Q4r5CxBmTxn/3745ptj15GWLXNNecWLuzOjvn3dWVKtWjBv3i+0bFne75CNyZZevXrx9ttvc8UVVzBq1CjOOisyExdGmiUok+sdOQLff3/sOtI338DBg27Q02bNYNgw17GhSRMbCNXkXsuWLaNChQqceuqp3HPPPdSrV4/Bgwf7HVZYWYIyuY6qa6ZLTUhz57px7gDq1IF+/dwZUvPm7qzJmNxs48aN3H///bzxxhv07duX5557jjp16pCYeld4HmYJyuQKmzcfu4b05ZduJHCA00+H//s/l5AuvhhOOcXfOI0JlX379jFq1CiefPJJkpOTufvuu7n//vv9DiuiLEGZqLR7t+vynZqUVq1y5WXKuOa61NG/q1WzG2RN3jRkyBDGjRtHx44defLJJ6lWrZrfIUWcJSgTFQ4dgkWLjnVsWLzYXVsqXNiNZdetm0tI9erZyN8m7/rmm28oWbIktWvX5p577qFjx440b97c77B8YwnK+CIlBVauPNZkN3++mz02Xz5o3Bjuu8+dJZ17rktSxuRl69at49577+X999+nU6dOTJ06lYoVK1KxYkW/Q/OVJSgTNqpuMNXU6Wb+/PNYQvrqK9i61ZWfdRbcdJM7Q7rwQjfOnTGxYPfu3YwYMYJnnnmGfPnyMXToUO6++26/w4oalqBMWKjCwIEwbhx06uTmSfr9d7esQgVo0+bYDLIx/keiiWETJkzgySefpGvXrowYMYJKlSr5HVJUsQRlQk7VjWE3bpx7/9577trRuHEuKZ11lnVsMLHryy+/RERo1aoVt99+O61bt6Zx48Z+hxWV7HKzCSlVuPNOGDMGbr/dXWvav9/dSDtgAJx9tiUnE5t+++03rrrqKi655BKefPJJAIoVK2bJKROWoEzIqMIdd8Czz7rmvWeeccmocGFLSiZ2JSYmMnDgQGrXrs28efMYOXIkM2fO9DusXCFsCUpEKovIXBH5WURWicgdXnkZEflCRFZ7z6W9chGRcSKyRkRWiEiDgH1199ZfLSLdA8obishKb5txktvHls/FVN0Z0vjxMGiQO4Oyb8MY+Pjjjxk/fjw33XQTq1ev5t5776WwdU0NSjjPoJKBu1S1JtAM6CciNYH7gDmqWgOY470HuByo4T1uAV4Al9CAoUBToAkwNDWpeev0DtiuTRjrYzKQkuKGF5owwV17GjXKkpOJXarKrFmzeOuttwC44YYb+Omnn5g4cSLly9vAxNkRtgSlqptU9Xvv9R7gF6Ai0B543VvtdeBq73V74A11FgGlRORU4DLgC1VNVNUdwBdAG2/ZSaq6SFUVeCNgXyZCUpPTCy/APffAU09ZcjKxa8WKFVxyySW0a9eO559/HlUlX758nH322X6HlitFpBefiFQB6gOLgfKquslbtBlI/ZOiIrAhYLMEryyz8oR0ytM7/i24szLKly9PfHx8juuSlJR0QtvnFsHUMyUFxow5k48/rkCXLutp02Yd8+ZFJr5Qse8z7/GjromJiUyePJnZs2dTvHhxBgwYQLt27ZgXxv8QMfGdqmpYH0BxYBlwjfd+Z5rlO7znj4ALAsrnAI2AwcCDAeUPeWWNgC8DypsDH2UVT8OGDfVEzJ0794S2zy2yqueRI6q9eqmC6v33q6akRCauULPvM+/xo67z5s3T/Pnz68CBA3X79u0ROWZe+k6BpZrO73VYz6BEpAAwDXhbVT/0iv8RkVNVdZPXTLfFK/8bqByweSWv7G+gZZryeK+8UjrrmzBLSYGbb4ZXX4WHHoJHHrFmPRNbVJX333+ftWvXMmTIEFq0aMH69eupUKGC36HlKeHsxSfAK8AvqjomYNFMILUnXnfgfwHl3bzefM2AXeqaAj8DLhWR0l7niEuBz7xlu0WkmXesbgH7MmFy5Aj06uWS09Ch8OijlpxMbPnuu++44IILuO6665g2bRqHDx8GsOQUBuHsxXc+0BW4WER+8B5tgZHAJSKyGmjtvQf4BPgDWAO8BPQFUNVEYDiwxHs86pXhrfOyt81aYHYY6xPzjhxxY+a99pqbpXbYMJ8DMiaCNm3aRNeuXWnatClr167lpZdeYvHixRSwaZrDJmxNfKq6AMjob+tW6ayvQL8M9jUZmJxO+VKg9gmEaYJ05Aj06AFvveXOmh56yO+IjImsffv2MXPmTIYMGcKQIUMoUaKE3yHleTYWn8lScjJ07w5TpsBjj8EDD/gdkTHhl5KSwptvvsk333zDpEmTqF69OgkJCZaYIsiGOjKZSk6Grl1dcnriCUtOJjbMnz+fxo0b06NHD3788Uf27NkDYMkpwixBmQwlJ8MNN8C778KTT7pJBI3JyzZt2kTHjh258MIL2bJlC2+99RYLFy60xOQTa+Iz6UpOFrp0gfffh6efdkMYGZPXFSlShKVLl/Loo49y1113UbRoUb9DimmWoMxxDh+G4cNrMn8+jB7tBn81Ji9KTk7m5ZdfZtq0aXz66aeUKlWK1atXW8+8KGFNfOZfDh2C666D+fNPZuxYS04m7/rss8+oV68et912G4cOHWL79u0AlpyiiCUoc9ShQ2569unToX//1Qwc6HdExoTetm3buOKKK2jTpg379+9n2rRpxMfHc8opp/gdmknDmvgMAAcPwrXXwqxZbk6n2rX/xs1gYkzekJKSQr58+ShZsiSJiYk8/fTTDBgwgEKFCvkdmsmAnUEZDh6Ejh1dcpowAfr39zsiY0Ln0KFDjBkzhlq1arFnzx4KFCjAt99+y+DBgy05RTlLUDHuwAG45hr46CM3p1Pfvn5HZExoqCozZsygVq1a3HXXXZx++uns2rULAJt8O3ewBBXDUpPTJ5/AxInQp4/fERkTGklJSbRq1YoOHTpQoEABZs+ezaeffkqlSpWy3thEDbsGFaP274cOHeCzz+Cll9z0GcbkdgcOHACgWLFi/Oc//+G5557j1ltvJX9++6nLjewMKgbt3w/t28Pnn8Mrr1hyMrnf/v37efzxx6lcuTKbN29GRJgyZQr9+vWz5JSLWYKKMarQuTN8+SVMnuymzzAmt1JV3nnnHc466ywefPBBmjdvbteX8hBLUDHmww9h5kw3fFGPHn5HY0zOHT58mBYtWtClSxfKli3L3Llz+fDDDylfvrzfoZkQyfDcV0SeUdWBIjIL0LTLVbVdWCMzIbdvnxsZok4duOMOv6MxJmcSExMpU6YMBQoU4MILL+Smm26iW7duxMXF+R2aCbHMGmff9J5HRSIQE35PPgl//QVvvgnWLG9ym6SkJEaOHMmYMWP46quvaNasGY899pjfYZkwyvBnSlWXec/zIheOCZetW2HUKDeUUYsWfkdjTPCOHDnC66+/zgMPPMDmzZvp0qULFStW9DssEwFZ/h0tIucDw4DTvfUFN0N7tfCGZkLl8GG45RbXe2/YML+jMSZ4qkqrVq2YN28ezZo1Y/r06TRr1szvsEyEBNPQ8wpwJ7AMOBLecEw4jBoFM2a4Jr6zz/Y7GmOytn79ek477TREhBtuuIE+ffpw3XXXWQ+9GBNML75dqjpbVbeo6vbUR9gjMyGRlOSmar/6arj7br+jMSZzO3fu5K677qJGjRpMnToVgN69e9O5c2dLTjEomDOouSLyNPAhcDC1UFW/D1tUJmTefBP27HEz4tr/bxOtkpOTmThxIkOHDiUxMZGbbrqJCy+80O+wjM+CSVBNvedGAWUKXBz6cEwo7d0LTz0FzZrBeef5HY0xGWvfvj2ffPIJF110EWPGjKFevXp+h2SiQJYJSlUvikQgJrRUXceI9evh1Vft7MlEn59//plq1apRuHBh+vXrxy233EK7du2sKc8cleE1KBG50XselN4jciGanHjsMZgyBR5+GFq29DsaY47ZunUrffv2pU6dOowbNw6Atm3b0r59e0tO5l8yO4Mq5j2XiEQgJnQOHIBnnoF27WDoUL+jMcY5ePAg48eP57HHHiMpKYnbbruNm2wwSJOJzG7Uneg9PxK5cEwozJgBiYkwYIA17Zno0aNHD959913atm3LqFGjONvueTBZCOZG3XGZLVfV20MXjgmFF16AqlXhYuvGYny2fPlyKlSoQPny5bn77rvp2bMnl156qd9hmVwimPugCgMNgNXeox5QEHfj7rKwRWZyZMYMmD8f+vWDfDZWvfHJxo0b6dmzJw0bNuSJJ54AoEGDBpacTLYE8xNWB2ipquNVdTzQCqinqq+r6usZbSQik0Vki4j8FFA2TET+FpEfvEfbgGVDRGSNiPwmIpcFlLfxytaIyH0B5VVFZLFXPlVECma38nmNKgwZAuecA/37+x2NiUX79u1j+PDhnHnmmUyZMoXBgwfzyCN2lcDkTDAJqjRwUsD74l5ZVl4D2qRTPlZV63mPTwBEpCbQGajlbfO8iMSJSBwwAbgcqAlc760L8KS3rzOAHUCvIGLK01asgF9/hdtug0KF/I7GxKJ77rmHhx9+mDZt2vDzzz/z1FNPUbJkSb/DMrlUMDfqjgSWi8hc3ECxLXCDx2ZKVeeLSJUg42gPvKuqB4F1IrIGaOItW6OqfwCIyLtAexH5BXejcBdvnde9mF4I8nh5kjcyDB07+huHiS3ffvstpUuX5uyzz+bee++lU6dOtLAh800IBHOj7qsiMptjI0rcq6qbT+CY/UWkG7AUuEtVdwAVgUUB6yR4ZQAb0pQ3BcoCO1U1OZ31jyMitwC3AJQvX574+PgcB5+UlHRC24fLypUnMWpUPWrW3MOqVctPeH/RWs9Qs3rm3ObNm5k4cSLx8fFcfPHFPPTQQ0eX+fmZ2neah6hq2B5AFeCngPflgThc0+LjwGSv/DngxoD1XgE6eo+XA8q7euuWw51ZpZZXDjxOZo+GDRvqiZg7d+4JbR8OCxeqliiheuaZqhs2hGaf0VjPcLB6Zt/u3bt1yJAhWqhQIS1SpIg+/PDDmpSUFLL9nyj7TnMfYKmm83sd0XlVVfWf1Nci8hLwkff2by/JpKrklZFB+XaglIjkV3cWFbh+TPniC+jQAf7zH/jqK7B53Ey4PfPMMzzxxBPceOONjBgxgsqVK2e9kTE5ENGOyCJyasDbDkBqD7+ZQGcRKSQiVYEawHfAEqCG12OvIK4jxUwv487FnWEBdAf+F4k6RJMPPoArroDq1WHBAktOJnzmzJnDvHlucu2BAweyePFi3nzzTUtOJqyCSlAicoGI9PRen+wlkay2eQdYCPxXRBJEpBfwlIisFJEVwEW4iRBR1VXAe8DPwKdAP1U94p0d9Qc+A34B3vPWBbgXGOR1qCiLaxaMGa+9BtddB02awLx57gzKmFD7/fffadeuHa1btz56P1OJEiVo0qRJFlsac+KCGUliKG6qjf8CrwIFgLeA8zPbTlWvT6c4wySiqo/jrkulLf8E+CSd8j841tMvpuzfDwMHwvnnw6efQtGifkdk8podO3bw6KOP8txzz1GkSBGeeOIJBg4c6HdYJsYEcw2qA1Af+B5AVTeKiA0g66P4eNi1Cx54wJKTCY/p06czbtw4evXqxfDhwylfvrzfIZkYFEyCOqSqKiIKICLFstrAhNfTT0ORIjYJoQkdVeXjjz8mKSmJzp070717d5o2bUqtWrX8Ds3EsGCuQb0nIhNxveZ6A18CL4U3LJORFStg7lzXxFfCzmNNCKxcuZJLL72Uq666ivHjx6OqxMXFWXIyvssyQanqKOADYBruOtTD6sbkMxF24ADceCOULw933ul3NCa327JlC7feeiv16tVj2bJlPPvss8THx9ukgSZqBNNJYhAwVVW/iEA8JhMPPggrV8JHH8HJJ/sdjcntVq5cyeTJk+nfvz9Dhw6lTJkyfodkzL8Ecw2qBPC5iCQCU4H3A2+4NZExdy6MGQN9+rh7n4zJLlXlgw8+4M8//+Tuu++mVatWrFu3jkqVKvkdmjHpCqaJ7xFVrQX0A04F5onIl2GPzBy1cyd07w5nnAGjRvkdjcmNfvvtN1q0aEGnTp147733SE52w1hacjLRLDsjSWwBNuOGGTolPOGY9PTvDxs3wltvQTHrQ2myYdOmTXTr1o0+ffrw+++/M2nSJBYtWkT+/BEd5cyYHAnmGlRfoBNwMvA+0FtVfw53YMaZOhXefhseecSNGmFMduzZs4fp06fTpUsXXnjhBU466aSsNzImSgTzZ1RlYKCq/hDmWEwaCQnumlPTpnD//X5HY3KDlJQU3nrrLRYvXsyECRM488wzSUhIYPny5ZacTK6TYROfiKT+a34a+EtEygQ+IhNe7EpJgR494NAhePNNsBYZk5UFCxbQtGlTunfvzpIlS9i7dy+AzWhrcq3MrkFN8Z6X4SYXXBbwWBrmuGLeuHEwZw6MHQs1avgdjYlmmzZt4tprr6V58+Zs3ryZN998k0WLFlHMLliaXC7Dv8tV9UrvOcuRy01o/fQT3HcfXHkl9O7tdzQm2hUqVIhFixbxyCOPMHjwYIraAI0mjwimk8QcVW2VVZkJjZQU6NoVTjoJXn4Z7KZ+k1ZycjKvvPIKM2bM4OOPP6ZMmTKsXbuWggUL+h2aMSGV2TWowt61pnIiUjrg+lMVwKbGC5NVq+CHH+Cxx9yQRsYE+uKLL6hfvz59+vRh7969bN++HcCSk8mTMrsGdSvuetNZ/Pv60/+A58IfWmx64w333KaNv3GY6LJ161auvPJKLr30Uvbt28cHH3zAvHnzONnGvDJ5WGbXoJ4FnhWRATY4bGS0awezZsHFF8Npp/kdjYkGKSkp5MuXj5IlS7Jlyxaeeuopbr/9dgoVKuR3aMaEXZbXoFR1vIjUBmoChQPK3whnYLFG1SUngA8/9DcW479Dhw7x/PPP88orrxztkbd48WIbadzElCyHOvKmfB/vPS4CngLahTmumLN5s3ueMAHstpXYparMnDmT2rVrc+edd1KhQgV27twJYMnJxJxgxuLrCLQCNqtqT6AuYD+hIbZqlXuuap36Y9bu3btp3bo17du3Jy4ujo8//phPP/2UihWtT5KJTcGMT7BfVVNEJNkbXWILbvgjE0Lffeeezz7b3zhM5B08eJBChQpRokQJypYty/jx47n11lspUKCA36EZ46tgEtRSESmFm+Z9GZAELAxnULHou+/g9NOhShW/IzGRsn//fsaOHcu4ceNYunQplSpV4r333vM7LGOiRjCdJPp6L18UkU+Bk1R1RXjDii07d7phja6/3u9ITCSoKlOnTuW+++5j/fr1tG/fnpSUFL/DMibqZJigRKRBZstU9fvwhBR7nn0WkpLgttv8jsSE26FDh2jVqhULFiygbt26vPrqq1x00UV+h2VMVMrsDGp0JssUuDjEscSknTvdgLBXXw316/sdjQmXnTt3UqpUKQoWLMj5559Pjx496NGjB3FxcX6HZkzUyuxGXfuzLgLGjoVdu2DYML8jMeGQlJTEk08+yZgxY5g3bx6NGjVi5MiRfodlTK4QzGCx3dIrtxt1T1xioktQ//d/ULeu39GYUEpJSeH111/ngQceYNOmTXTu3JnyNriiMdkSTC++xgGvC+PuifoesAR1gsaMcdee7Owpb1FVLrroIubPn0/Tpk2ZNm0a5557rt9hGZPrZHmjrqoOCHj0BhoAxbPaTkQmi8gWEfkpoKyMiHwhIqu959JeuYjIOBFZIyIrAjtoiEh3b/3VItI9oLyhiKz0thknuew2e1V46SV37al2bb+jMaGwYcMGVBURoXPnzrz99tssXLjQkpMxORTMSBJp7QWCGe/gNSDtmNz3AXNUtQYwx3sPcDlQw3vcArwALqEBQ4GmQBNgaGpS89bpHbBdrhr/e9Mm2LIFrANX7rdz504GDx5M9erV+dAbSPG2226jS5cuNjyRMScgmGtQs3C99sAltJpAlncTqup8b+6oQO2Blt7r14F44F6v/A1VVWCRiJQSkVO9db9Q1UQvli+ANiISj7sfa5FX/gZwNTA7q7iixULvVucGGXbmN9EuOTmZSZMmMXToULZv307Pnj0577zz/A7LmDwjmGtQowJeJwPrVTUhh8crr6qbvNebgdSrxhWBDQHrJXhlmZUnpFOeLhG5BXdmRvny5YmPj89h+K5X1olsn+qNN2pQpEh59u//hvh4zXqDCAtVPaPdidTz3nvv5bvvvqNevXqMGDGCGjVq8Ntvv/Hbb7+FNsgQiJXvE2KnrrFQz2BGkpgH4I3Dl997XSb1rCanVFVFJCK/zKo6CZgE0KhRI23ZsmWO9xUfH8+JbA+uaW/OHGjVClq3vvCE9hUuoahnbpDdev7yyy9Uq1aNQoUK8eCDD3LkyBHat28f9U15sfJ9QuzUNRbqGcx0G7eIyGZgBbAUNx7f0hwe7x+v6Q7veYtX/jf/HoC2kleWWXmldMpzhR9/hL17YdAgvyMxwdq2bRv9+vXjnHPOYcKECQBcddVVXH311VGfnIzJrYLpJHE3UFtVq6hqNVWtqqrVcni8mUBqT7zuuOnjU8u7eb35mgG7vKbAz4BLRaS01zniUuAzb9luEWnm9d7rFrCvqJfaAlS9ur9xmKwdOnSI0aNHc8YZZzBx4kT69OlD165d/Q7LmJgQzDWotcC+7O5YRN7BdXIoJyIJuN54I4H3RKQXsB7o5K3+CdAWWOMdqyeAqiaKyHBgibfeowFNi31xPQWL4DpH5JoOEm+/7abVqGyTlkS9rl278t5773H55ZczatQoatas6XdIxsSMYBLUEOBbEVkMHEwtVNXbM9tIVTMam7tVOusq0C+D/UwGJqdTvhTIdXcQbd4MixbB8OFgLUPRafny5VSuXJly5coxePBgevbsSZs2ueouBmPyhGCa+CYCXwGLcNefUh8mBz7+2D1fdZW/cZjjbdq0iZtuuomGDRvyxBNPANC4cWNLTsb4JJgzqAKqapfzQ0AVRo2CWrWgTh2/ozGp9u/fz+jRoxk5ciSHDh3irrvu4oEHHvA7LGNiXjAJarZ3H9Es/t3Ed0LdzGPR0qXw66/w8svWvBdNBg0axIsvvsg111zDU089RXXrvWJMVAgmQaVeSxoSUKZATnvyxawPPoACBdzo5cZfCxcupGzZsoC74bZz585ceGF03pNmTKwKZrDYquk8LDnlwBdfwLnnQqlSfkcSu9avX0/nzp0577zzGD58OABVqlSx5GRMFLL5oCLkxx9h+XJ4+mm/I4lNe/bsYeTIkYwePRoR4aGHHuKee+5h6dKc3nNujAk3mw8qQsaMgRIloFcvvyOJTaNGjWLEiBHccMMNPPHEE1S2m9CMiXrBjMU3IPC9iJQC3g1XQHlRSgrMng3t20Pp0lmvb0Jj7ty5FChQgAsuuIBBgwbRtm1bmjZt6ndYxpgghXM+KOP58UfYuhVat/Y7ktiwevVqrr76ai6++GJGjBgBQMmSJS05GZPLhG0+KHPMG29AXBy0bet3JHnbjh07GD58OM899xyFChVixIgR3HnnnX6HZYzJoUjPBxWTpk+Hdu3g5JP9jiRve//993nmmWfo1asXw4cP5z//+Y/fIRljTkCGCUpEzsBNMDgvTfn5IlJIVdeGPbo8YMMGWL8eBgzIel2TParK7Nmz2bt3L9deey033XQT5513HrVr57ohGo0x6cjsGtQzwO50ynd7y0wQpk1zz+3b+xtHXvPTTz9x2WWXccUVV/Dss8+iquTPn9+SkzF5SGYJqryqrkxb6JVVCVtEeciRI/DCC27cvTPO8DuavGHLli306dOHunXrsmTJEsaOHctXX31lkwYakwdldg2qVCbLioQ4jjzpgw/g99/dswmN5cuX8/LLL9OvXz+GDh16dLgiY0zek1mCWioivVX1pcBCEbkZm24jS6pu1Ihq1aBDB7+jyb1UlQ8//JC//vqLO++8k8suu4w//viD0047ze/QjDFhllmCGghMF5EbOJaQGgEFAfvJzcLatbBsGYwfD/lycreZYenSpQwaNIivv/6aRo0acfvttxMXF2fJyZgYkeFPp6r+o6rnAY8Af3qPR1T1XFXdHJnwcq/vvnPP55/vbxy50aZNm+jevTuNGzfm119/5cUXX2ThwoXExcX5HZoxJoKCGepoLjA3ArHkKZ99BmXL2sSEOZGYmMi0adO49957GTJkCCVLlvQ7JGOMD4K5UdfkwHffubMn+6M/aykpKbz99tssXbqUZ599llq1apGQkEApm5fEmJhmV0fC4O+/4bffoFEjvyOJfgsWLKBZs2Z069aNb7/9ln379gFYcjLGWIIKh1mzXC++a6/1O5LotXHjRjp16kTz5s3ZuHEjb7zxBosXL6Zo0aJ+h2aMiRLWxBcGCxe6cff++1+/I4leBQoUYMGCBQwbNozBgwdTrFgxv0MyxkQZS1AhtncvfPwxXHIJ2OAGxxw5coRXXnmFjz76iBkzZnDyySezbt06ChUq5HdoxpgoZU18Ifbkk7B9uw0OG2jOnDnUr1+fW2+9lR07drBjxw4AS07GmExZggqhgwfd2HtXXw3nned3NP7bsmULV111Fa1bt2bPnj28//77zJ8/34YnMsYExZr4QqhKFdi2Dfr29TsSf6kqIkLJkiVJSEhg5MiR3HHHHRQuXNjv0IwxuYglqBA5dAg2e+NrxOrU7ocPH+b555/n1Vdf5dtvv6Vo0aIsW7aMfDbWkzEmB+yXI0S++so9z5oVe50jVJVZs2ZRu3ZtBg4cyCmnnMLOnTsBLDkZY3LMl18PEflTRFaKyA8istQrKyMiX4jIau+5tFcuIjJORNaIyAoRaRCwn+7e+qtFpLsfdUk1fToULx57Z0+7du3ikksuoV27dogIH330EZ999hkVKlTwOzRjTC7n55+3F6lqPVVNHW/hPmCOqtYA5njvAS4HaniPW4AXwCU0YCjQFGgCDE1NapF25AjMmAFt20KsXGY5dOgQACeddBIlSpRg3LhxrFy5kiuuuMImDzTGhEQ0tb+0B173Xr8OXB1Q/oY6i4BSInIqcBnwhaomquoO4AugTYRjBtyNuVu2wDXX+HH0yDpw4AAjR46kSpUqbNq0CRFh+vTpDBgwgAIFCvgdnjEmD/Grk4QCn4uIAhNVdRJuivlN3vLNQHnvdUVgQ8C2CV5ZRuXHEZFbcGdflC9fnvj4+BwHnpSUdNz2EyZUp0CBihQv/g3x8UdyvO9okraeqkp8fDyTJk1i8+bNnHfeecyfP5/y5ctnvJNcIL3vMy+KlXpC7NQ1JuqpqhF/ABW951OAH4EWwM406+zwnj8CLggon4ObOHEw8GBA+UPA4KyO3bBhQz0Rc+fO/df7lBTVqlVV27Y9od1GncB6Hjx4UJs3b66A1qlTR+fMmeNfYCGW9vvMq2KlnqqxU9e8VE9gqabze+1LE5+q/u09bwGm464h/eM13eE9b/FW/xuoHLB5Ja8so/KI+uUXWLcO2reP9JHDb9euXQAULFiQxo0b89JLL/H9999z8cUX+xyZMSYWRDxBiUgxESmR+hq4FPgJmAmk9sTrDvzPez0T6Ob15msG7FLXFPgZcKmIlPY6R1zqlUXUypXuuVmzSB85fPbu3curr75KpUqV+OGHHwAYPXo0N998s81qa4yJGD+uQZUHpns9vfIDU1T1UxFZArwnIr2A9UAnb/1PgLbAGmAf0BNAVRNFZDiwxFvvUVVNjFw1nD/+cM/VqkX6yKGXkpLCm2++yf3338/GjRu57rrrKFOmjN9hGWNiVMQTlKr+AdRNp3w70CqdcgX6ZbCvycDkUMeYHb//Dv/5j7sHKjdLSUmhZcuWfP311zRp0oQhQ4bQv39/v8MyxsSwaOpmniutWgW1avkdRc79/fff7mJkvnxce+21vPXWWyxcuJDatWv7HZoxJsZZgjoBqvDrr3D22X5Hkn27du3i7rvvplq1asyaNQuAAQMGcMMNN9jwRMaYqGCDxZ6AzZthz57cNXNucnIyL7/8Mg899BDbt2+ne/fuNGrUKOsNjTEmwixBnYA1a9xzjRr+xpEdV1xxBZ9//jktWrRg7NixNGjQIOuNjDGZOnz4MAkJCRw4cCBixyxZsiS//PJLxI4XCoULF6ZSpUpBjzpjCeoErF/vnqtU8TWMLP32229UrVqVggUL0qdPH2699VY6dOhgY+YZEyIJCQmUKFGCKlWqROz/1Z49eyhRokREjhUKqsr27dtJSEigatWqQW1jFxtOwLp17vm00/yNIyPbtm1jwIAB1KpVi4kTJwLQoUMHrrnmGktOxoTQgQMHKFu2rP2/yoSIULZs2WydZdoZ1An46iuoXRuKFPE7kn87dOgQEyZM4NFHH2XPnj3ccsstdO7c2e+wjMnTLDllLbufkSWoHFKF77+Hrl39juR4Xbp0Ydq0aVx22WWMHj2aWrm5H7wxJmZZE18O7dgBu3dHzwgSP/zwA4mJbiCNwYMH88knn/Dpp59acjLGpOu11147ejP+iy++yBtvvAHAr7/+Sr169ahfvz5r165lypQpvsVoCSqH1q51z0Fe6wubTZs20atXLxo0aMBTTz0FQLNmzbj88sv9DcwYk2v06dOHbt26ATBjxgw6duzI8uXL2bBhg68Jypr4cih1kFi/BlzYv38/Y8eOZcSIERw6dIhBgwZx3333Zb2hMSasBg4Eb4zlkKlXD555JvN1/vzzT6688kp++uknAEaNGnV0zqi6desyb948kpOTmTx5Mk2aNPnXtsOGDaN48eLUrFmTZ555hri4OObMmcP+/fv55ZdfqFevHt27d+fOO+8MbcWyYAkqh1asgKJFoXp1f45/xx138NJLL9GhQweeeuopzjjjDH8CMcZEvX379vHDDz8wf/58brrppqNJLK22bdvSp08fihcvzuDBg4mPj2fUqFF89NFHEY7YsQSVQz/+COecA5EcFWjRokWcfPLJVK9enXvvvZcuXbrQsmXLyAVgjMlSVmc6frj++usBaNGiBbt372bnzp3+BhQkuwaVA6ruDKpOncgc76+//qJLly6ce+65DB8+HIDq1atbcjLGHJU/f35SUlKOvg+83yht9+7c0iXeElQObNgAiYlQ97hJQ0IrKSmJBx98kP/+979Mnz6dBx54gPHjx4f3oMaYXKl8+fJs2bKF7du3c/DgwX81y02dOhWABQsWULJkSUqWLBnUPkuUKMGePXvCEm8wrIkvB1591T23Om72qtAaOXIkjz/+OF26dOGJJ57gtGgdssIY47sCBQrw8MMP06RJEypWrMhZZ511dFnhwoWpX78+hw8fZvLk4KfQq1OnDnFxcdStW5cePXpEvJMEqhpTj4YNG+qJmDt3rp55puqll57QbjLd/8KFC1VVNTEx8ejrSJs7d64vx400q2fe40ddf/7554gfc/fu3UGtd+GFF+qSJUvCHE3w0vusgKWazu+1NfFl08GD+Vi9Gs49N7T7XbNmDR06dOCiiy7i8ccfB6B06dI0a9YstAcyxphcwpr4smnjxsKohm4OqJ07dzJ8+HDGjx9PwYIFefzxxyN/Gm2MybPi4+P9DiHHLEFl0/bthQCoVCk0+5syZQpjx46lZ8+ePPbYY5x66qmh2bExxuRylqCyadu2ggBUqJDzfcyePZuDBw9y9dVX07t3by644ALqRKrPujHG5BJ2DSqbtm1zZ1A5OdFZtWoVl19+OW3btmXMmDGoKgUKFLDkZIwx6bAElU0bNhSlQgU3zFGwtm7dSt++falbty6LFi1i9OjRfPnll7nmZjljjPGDJahs+uuvopx9dva2Wbx4MZMmTeK2225j9erVDBo0iIIFC4YnQGOMCYEqVaqwbds2X2Owa1DZcOgQrF1bnCuvzHw9VWX69Ols3LiR/v37c8UVV7B69Wqq+j03hzEmJhy9jyiSg4WGgSWobPjtNzh8OB8NGmS8zvfff8+gQYOYN28ejRo14rbbbiMuLs6SkzExJL1xMjt16kTfvn3Zt28fbdu2PW55jx496NGjB9u2baNjx47/WhZMV/E///yTyy67jKZNm7Js2TKaNGnCypUr2b9/Px07duSRRx4B3JlR9+7dmTVrFocPH+b999/nrLPOYvv27Vx//fX8/fffnHvuubj7Z50xY8YcHYHi5ptvZuDAgfz555+0adOGZs2a8e2339K4cWN69uzJ0KFD2bJlC2+//fZx03pkV+5OrxH288/uOb1Jajdt2kTPnj1p1KgRq1at4oUXXmDhwoXExcVFNkhjTMxavXo1ffv2ZdWqVYwePZqlS5eyYsUK5s2bx4oVK46uV65cOb7//ntuu+02Ro0aBcAjjzzCBRdcwKpVq+jQoQN//fUXAMuWLePVV19l8eLFLFq0iJdeeonly5cDboCBu+66i19//ZVff/2VKVOmsGDBAkaNGsWIESNOuD52BpUNv/4KIkqNGsd3bti6dStTp05l8ODBPPDAA0EPxmiMyXsyO+MpWrRopsvLlSuX45trTz/99KOjz7z33ntMmjSJ5ORkNm3axM8//3y0x/A111wDQMOGDfnwww8BmD9//tHXV1xxBaVLlwbcALMdOnSgWLFiR7f9+uuvadeuHVWrVuWcc84BoFatWrRq1QoR4ZxzzuHPP//MUR0C5foEJSJtgGeBOOBlVR0ZrmOtXQsnn3yQIkUKk5KSwjvvvMPy5csZNWoUderUISEhgTJlyoTr8MYYk6nUJLJu3TpGjRrFkiVLKF26ND169PjX9BuFCrnbZeLi4khOTs7x8VL3A5AvX76j7/Ply3dC+z26zxPeg49EJA6YAFwO1ASuF5Ga4TremjVQocJ+vv32W84991xuvPFG4uPj2b9/P4AlJ2NMVNi9ezfFihWjZMmS/PPPP8yePTvLbVq0aMGUKVMAN5jAjh07AGjevDkzZsxg37597N27l+nTp9O8efOwxp8qVycooAmwRlX/UNVDwLtA+3AdrHfvjRw50oXzzz+fhIQEXnvtNb777juKFCkSrkMaY0y21a1bl/r163PWWWfRpYv7zcrK0KFDmT9/PrVq1eLDDz88Or1PgwYN6NGjB02aNKFp06bcfPPN1K9fP9xVcNIb4jy3PICOuGa91Pddgecy2+ZEptvYtGmTli5dWh9++GFNSkrK8X5yg1iZnsHqmffYdBvRLTvTbYgGdCXMbUSkI9BGVW/23ncFmqpq/zTr3QLcAlC+fPmG7777bo6PmZiYGBNNeUlJSRQvXtzvMMLO6pn3+FHXkiVLcsYZZ0T0mEeOHMmVvYTXrFnDrl27/lV20UUXLVPVRmnXze2dJP4GKge8r+SV/YuqTgImATRq1EjTu0chWPHx8ene45DXWD3zllipJ/hT119++YUSJUpE9Jh79uyJ+DFDIXV232Dk9mtQS4AaIlJVRAoCnYGZPsdkjIlBubk1KlKy+xnl6gSlqslAf+Az4BfgPVVd5W9UxphYU7hwYbZv325JKhOqyvbt2ylcuHDQ2+T2Jj5U9RPgE7/jMMbErkqVKpGQkMDWrVsjdswDBw5k68c+GhQuXJhK2ZjtNdcnKGOM8VuBAgUiPt5mfHx85Lp7+yRXN/EZY4zJuyxBGWOMiUqWoIwxxkSlXH2jbk6IyFZg/Qnsohzg7zSTkWH1zFtipZ4QO3XNS/U8XVVPTlsYcwnqRInI0vTueM5rrJ55S6zUE2KnrrFQT2viM8YYE5UsQRljjIlKlqCyb5LfAUSI1TNviZV6QuzUNc/X065BGWOMiUp2BmWMMSYqWYIyxhgTlSxBpUNE2ojIbyKyRkTuS2d5IRGZ6i1fLCJVfAgzJIKoaw8R2SoiP3iPm/2I80SIyGQR2SIiP2WwXERknPcZrBCRBpGOMRSCqGdLEdkV8F0+HOkYQ0FEKovIXBH5WURWicgd6ayTV77TYOqaJ77XdKU3zW4sP4A4YC1QDSgI/AjUTLNOX+BF73VnYKrfcYexrj2A5/yO9QTr2QJoAPyUwfK2wGxAgGbAYr9jDlM9WwIf+R1nCOp5KtDAe10C+D2df7d55TsNpq554ntN72FnUMdrAqxR1T9U9RDwLtA+zTrtgde91x8ArUREIhhjqART11xPVecDiZms0h54Q51FQCkROTUy0YVOEPXME1R1k6p+773eg5sLrmKa1fLKdxpMXfMsS1DHqwhsCHifwPH/II6uo27SxF1A2YhEF1rB1BXg/7xmkg9EpHJkQouoYD+HvOBcEflRRGaLSC2/gzlRXvN6fWBxmkV57jvNpK6Qx77XVJagTFZmAVVUtQ7wBcfOHE3u8z1uzLO6wHhghr/hnBgRKQ5MAwaq6m6/4wmnLOqap77XQJagjvc3EHiWUMkrS3cdEckPlAS2RyS60Mqyrqq6XVUPem9fBhpGKLZICuY7z/VUdbeqJnmvPwEKiEg5n8PKEREpgPvBfltVP0xnlTzznWZV17z0vaZlCep4S4AaIlJVRAriOkHMTLPOTKC797oj8JV6VytzmSzrmqbdvh2uDTyvmQl083p+NQN2qeomv4MKNRH5T+q1UhFpgvv/n+v+sPLq8Arwi6qOyWC1PPGdBlPXvPK9psemfE9DVZNFpD/wGa6X22RVXSUijwJLVXUm7h/MmyKyBndRurN/EedckHW9XUTaAcm4uvbwLeAcEpF3cD2dyolIAjAUKACgqi8Cn+B6fa0B9gE9/Yn0xARRz47AbSKSDOwHOufSP6zOB7oCK0XkB6/sfuA0yFvfKcHVNa98r8exoY6MMcZEJWviM8YYE5UsQRljjIlKlqCMMcZEJUtQxhhjopIlKGOMMTmS1QDF6azfKWDg2ylZrW8JypgAIqIiMjrg/WARGRaifb8mIh1DtK/F3sjVf8m/R5uvEor9p3O8+3OwTQ8ReS4c8Zio8RrQJpgVRaQGMAQ4X1VrAQOz2sYSlDH/dhC4JtruxPdGLDlKVZuqaj3gYdxo+vW8x5/Z2U82ZDtBmbwvvQGKRaS6iHwqIstE5GsROctb1BuYoKo7vG23ZLV/S1DG/FsyMAm4M+2CtGdAIpLkPbcUkXki8j8R+UNERorIDSLynYisFJHqAbtpLSJLReR3EbnS2z5ORJ4WkSXeoLy3Buz3axGZCfycVeAicpV3ZrVcRL4UkfJe+TAReVNEvsHdYH6yiHzhNbO8LCLrUxOyiNzoxf2DiEz0YhsJFPHK3s5oPa+8p1e373A3mZrYMwkYoKoNgcHA8175mcCZIvKNiCwSkSzPvCxBGXO8CcANIlIyG9vUBfoAZ+Pu/D9TVZvgxi8cELBeFdw0J1cAL4pIYaAXbiiexkBjoLeIVPXWbwDcoapnBhHDAqCZqtbHTZ1yT8CymkBrVb0eN8LEV14zywd4oxKIyNnAdbgmmHrAEeAGVb0P2O+dod2Q0XrihsV6BJeYLvCOaWKIuEFtzwPe90a+mIib0wrcyEU1cKOdXA+8JCKlMtufDXVkTBqqultE3gBuxw0dE4wlqWO9icha4HOvfCVwUcB676lqCrBaRP4AzgIuBeoEnJ2VxP1HPgR8p6rrgoyhEjDVSxQFgcDtZqpqal0uADp4df1URHZ45a1wgwEv8YZ2KwKk1wyT0XpNgXhV3ep9DlNxfzWb2JEP2On94ZJWAm7iyMPAOhH5HffvfElmOzPGHO8Z3JlNsYCyZLz/MyKSD5cEUh0MeJ0S8D6Ff/8hmHZsMcXN+jog4DpSVVVNTXB7sxHzeNzsx+cAtwKFA5YFsx8BXg+I47+qOuwE1jMxxpsKZJ2IXAtusFsRqestnoE7e8JrUj4T+COz/VmCMiYdqpoIvIdLUqn+5Nh0I+3wBmLNpmtFJJ93Xaoa8BtusN7bxE2rgIicKSLFMttJBkpybEqJ7pms9w3QyTvWpUBpr3wO0FFETvGWlRGR071lh1Pjy2S9xcCFIlLWW/faHNTB5CLiBiheCPxXRBJEpBdwA9BLRH4EVnFslu7PgO0i8jMwF7hbVTMddd2a+IzJ2Gigf8D7l4D/ef/xPiV7Zzep/gK+A04C+qjqARF5GXdt6ntxbWZbgatzsO9huLb/HcBXQNUM1nsEeEdEuuJ+XDYDe1R1m4g8CHzunSEeBvoB63EXvleIyPfedajj1lPVReK65C8EdgI/5KAOJhfxrmmm57gOEN4I64O8R1BsNHNjYoyIFAKOeNOtnAu8kME1A2N8ZWdQxsSe04D3vLOfQ7j7U4yJOnYGZYwxJipZJwljjDFRyRKUMcaYqGQJyhhjTFSyBGWMMSYqWYIyxhgTlf4ftSK87gMb39gAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "perf_h2o.plot_uplift(metric=\"qini\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAABGhElEQVR4nO3debyN1f7A8c/XLGTuVAjlqFTmKRqkQki5l+ImQ6UkFaUM3UyVJvwqmiVzpXsLdbmSnFzKrBIRypiZMp7J+f7+WM9hO86wD3ufPZzv+/V6Xnuf9Uzf52z29zxrrWctUVWMMcaYcJMn1AEYY4wx6bEEZYwxJixZgjLGGBOWLEEZY4wJS5agjDHGhCVLUMYYY8KSJSiTa4nIEBGZHOo4/CEiXUVkoc/PR0TkUu99YRH5QkT+EpFPQxdleBCR2SLSJdDbmpxnCcrkOBH5h4gs975kd3pfEteFOq7MiMh4EXn+HPaPE5EH0pQ1EZHtZ3M8VS2qqr95P7YDYoDSqtren1jFeUxEfhaRoyKyXUQ+FZFrziaecKKqt6nqhEBva3KeJSiTo0TkCeA1YDjuS/US4C3gjhCGFekqAr+qanI29nkdeBx4DCgFVAWmA62ye3IRyZfdfYzxi6raYkuOLEBx4AjQPpNtxgPP+/zcBNju8/Nm4CngJ+Ao8AEu0c0GDgNfAyXT29dn/1u890OAyT7rPgV2AX8BC4CrvPIHgSQg0Yv/C6/8YuDfwF7gd+CxTK4rDnggTVnaa1NcwvgN2Ae8CuTx1nUFFqbZtgow1IsryYvtofRiTXPeWOAEUN/feDM4/yPABu/a3wZGpDnGDOCJ7P6u/Ph3lBcY4f2OfvPiUCBf2thT4/a2P+id+7bMPhdbwmexOyiTk64FCgGfn+Nx/g7civur/3ZcchoIlMXVCjx2lsedjfvyvgBYCUwBUNX3vPevqKtau11E8gBfAD8C5YCbgd4i0vxsL8rTFqgL1MbdVd6X2caqOhh3N/qJF9u7aWNNZ7ebcYlx6TnGeifQAKgGfATcLSICICIlgWbAx0H4XXUHWgO1cL+rdlls3wBYD5QBXgE+SI3ThDdLUCYnlQb2afaqotIzWlV3q+oO4H/AElVdparxuORX62wOqqrjVPWwqibg7q5qiEjxDDavB5RV1WGqmqiuPeh9oMPZnNvHy6p6QFW34qpCO57j8dJTGtgZgOO86MV6HPc5KHC9t64d8L2q/kHgf1d3Aa+p6jZVPQC8mMX2W1T1fVU9AUwALsLddZswZ3XHJiftB8qISL5zTFK7fd4fT+fnotk9oIjkBV4A2uPuxFK8VWVwVX5pVQQuFpE/fcry4r6o05MM5E9Tlh9XHedrm8/7LbiqsUDbj/uSPlcnY1VVFZGPcQl1AfAPILWHpN+/KxG5BFjrc9z0PsuLOfP3lJldPsc75t08ZfvfiMl5dgdlctL3QAKuaigjR4HzfH6+8BzOd9qxvCRUNoNt/4GrUrsF11ZWKXU37zXtsP/bgN9VtYTPUkxVW2Zw/K0+x0xVmTO/XCv4vL8E+COD42UmqykK5gHlRaRuJtv48zmkPc9HQDsRqYirVvu3V+7370pVt3pVk0UzSE7g7v7S/p5MFLIEZXKMqv4FDALeFJE7ReQ8EckvIreJyCveZj8ALUWklIhcCPQ+h1P+ChQSkVYikh/4J1Awg22L4ZLnftwX8/A063cDl/r8vBQ4LCL9vOeQ8orI1SJSL4PjfwJ0E5H6XhfvqkAf4OM02z0lIiVFpAKul90nfl5rZrGeRlU34HpOfuR1dS8gIoVEpIOI9Pc2+wH4m/cZVQHuz+qkqroK13FhLDBHVf/0VmX3d5WVacBjIlLea+vqn9UOJjJZgjI5SlVHAk/gksVe3F/XvXBdnAEm4RrTNwNfcXZf0Knn+gvoifvC3IG7K8jouaOJuLuZHbgqpsVp1n8AVBORP0Vkutee0RqoiesZlvrFnG6blarOwX2RfoirMpyFaw95L82mM4AVuATxH++82XVarBls8xgwBngT+BPYhOug8YW3/v9wPQF3e3FO8fPcU3F3oVNTC7L7u/LD+8Ac3L+TlcBnZ3kcE+ZE1SYsNCYciIgCsaq6MdSxRBIRqYRLfPkD0AHHhBG7gzLGGBOWLEEZY4wJS1bFZ4wxJizZHZQxxpiwZAnKGGNMWLIEZaKaN83FQREpmKYsw6kvRKSSiGjaUbrTTmMhIheJyAfelCGHRWSdiAwVkSLZjPE5EVktIskiMiTNuoHetCSpy3ERSRGRMt76ciIyQ0QOeFNm9Eizf1MRWSkih0TkNxF5MIMYxnnXXCWTODM9loiUFZGp4ualOigiU9Ksv8XbP3V6j7t81tUUkRUicsx7renv789EL0tQJmp53Y+vx4140CbAxy6FGxmjMHCtqhbDDWBbArgsm4fbCDyNe+7pNKo6PM3ICi8Dcaq6z9tkMq6LdQxuqozhInKTF2N+3NiE7+KeObobGCUiNdJcy3VZxeznsT7DDSt0CW7A3RE++1fDPRv1jLd/DdzzXohIAdzzX5OBkrjnrmZ45SYXswRlolln3AO344FAz5r6BG56j06quhnAG7z0cVX9KTsHUtUJqpo6XUiGxA0i1xn3BY6IFMVN2fGCqiap6o/Avzg1Anop4HxgkjrLgF9wo4+nHjMfMBp4NIswMz2WiDTDDT/0lKr+5cWzymf/fwLvqupsVU1W1f2quslb1wQ3Luhrqpqgqm/ghphqmkVMJspZgjLRrDNuBIQpQHMRCeQI1rcAn6lqSkYbiMhP3mgO6S1vncU5r8fdmaSOcSdpXlPfXw2gqrtx4+N184YXuhY3cOtCn+37AAuySqp+HKshbkqLCSKyX0SWiciNPodoCOBVZe4UkcneXSjAVcBPenqX4p+8cpOLWYIyUcmrtqoITFPVFbihfP4RwFNkOWWFqlZPM0Cq79LzLM7ZBfiXqh7xjn8YWAQ8642lVxs3V5bvIK8f4cY/TMCNHv6Mqm4DEDfe30Peen9keCygPG7+p/m4gWVH4qrpyvisv9eLLxZXNTraW1eUM0eM/ws3PqLJxSxBmWjVBfjKp61mKqeq+bKa+iLZpyyjbQI1ZYVfROQ83FQgE9Ksugc3Kvo23Ky2k/HGGxSRK3CD0XYGCuDuSJ4WkdRp3V8DhnljFmZ1/qyOdRzYrKofeNV7H3sxNfZZ/6Gq/uol2OFA6mjmR3DVh77OJ4sqTxP9LEGZqCMihXGT2t0oIrtEZBeuKquG16if1dQXO3GJKLNtvgbaipstNqM41qTpgee7vJPNy2oLHMBNUX6Sqm5R1daqWlZVG+Dmr0qdKfdq4FdVnaOqKaq6HtcR4zZv/c3Aqz6/I4DvRSS9O82sjvUTZ06/kbbKTjNYtwao7rWxparulZvcTMNg3nlbbAnkgps07wCuN9mFPssCXNVTc2APUB/XZlMV1+Dfw+cYH+F6pZXG3Tl1xI36HeOtL4UbcX0SUNErKweMAqpnM978QCHcXd7z3vu8abb5Cne3k3bfK3FVYQWATriRwst66y7D3Z009a7zMlyPwQe99Rek+f0orq2ocDrnyepYpYCDuLvUvLgZdQ8AZbz19+F6G16Kq4KchutwgRf7Ftz0IgVxo9tvAQqE+t+SLaFdQh6ALbYEegH+C4xMp/wuXDfofN4X5hrgkPdF2x/I47NtSU5N03EQ19bTOM3xLgbGecc8DKwDBgPnZTPe8V5y8F26+qwvh6t2rJLOvr1x05YcxXVYqJvONf/sxbcd1009TwZxqO85gNnAQH+PhevEsdpLZMuB69Mcf6gX615cYi/ps64Wrtv5cdwUGrVC/e/IltAvNhafMcaYsGRtUMYYY8KSJShjjDFhyRKUMcaYsGQJyhhjTFjKl/Um0aVMmTJaqVIlv7c/evQoRYpka3DqiGXXGp3sWqNPtF3nihUr9qlq2bTluS5BVapUieXLl/u9fVxcHE2aNAleQGHErjU62bVGn2i7ThHZkl65VfEZY4wJS0FLUCJSQUTmi8hab8iXx73yISKyQ0R+8JaWPvsMEJGNIrJeRJr7lLfwyjaKSH+f8soissQr/8TmjzHGmOgRzDuoZOBJVa2GGz7lEW/SMoD/U9Wa3jILTk5o1gE3CGUL4C1vWP+8wJu4Mb+qAR19jvOyd6wquKf97w/i9RhjjMlBQUtQqrpTVVd67w/jxjorl8kudwAfq5uw7Hfc8DP1vWWjqv6mqom4EZXv8AaWbIqboA3cKM93BuVijDHG5Lgc6SThTb1dC1iCG36/l4h0xo3X9aSqHsQlr8U+u23nVELblqa8AW4Qzz9VNTmd7dOe/0HgQYCYmBji4uL8jv3IkSPZ2j6S2bVGJ7vW6JNbrjPoCcqblvrfQG9VPSQibwPP4QamfA43uvR9mRzinKnqe8B7AHXr1tXs9H6Jtt4ymbFrjU52rdEnt1xnUHvxiUh+XHKaoqqfgZs6WlVPqJsq+31cFR64UaMr+Oxe3ivLqHw/UEJE8qUpN8YYkwMSEmBHEL91g9mLT4APgF9UdZRPue8spG1xw/cDzAQ6iEhBEamMmxZ6KbAMiPV67BXAdaSYqW4Y9vm4eWfAzUMzI1jXY4wx5nRNmtzHpZd+x4YNwTl+MKv4GgP3AqtF5AevbCCuF15NXBXfZuAhAFVdIyLTgLW4HoCPqOoJABHpBczBTYQ2TlVTZ9rsB3wsIs8Dq3AJ0RhjTJAcPHiQYsWKkZiYj19+aULlyluoUqVRUM4VtASlqgtxM2+mNSuTfV4AXkinfFZ6+6nqb5yqIjTGGBMkycnJvPvuuwwaNIgXXniBI0d68NdfnZk5EyS9b/oAyHVDHRljjMmer776ij59+rB27VpuuukmatRoxO23Q7NmcMMNwTuvDXVkjDEmQ3369KF58+YkJCQwffp05s2bx9dfV2f/fnj++eCe2+6gjDHGnObgwYPky5ePYsWK0bp1a8qVK8ejjz5KwYIFOXAARoyAO+6AevWCG4fdQRljjAFcO9Obb75JlSpVeO655wC4+eab6du3LwULFgRccjp8GLzVQWUJyhhjDHPmzKFGjRr06tWLGjVq0KlTpzO22b0bXn8dOnSAa64JfkyWoIwxJpcbNmwYLVq0ICEhgc8//5x58+ZRvXr1M7Z76SX3cO6QITkTl7VBGWNMLnTgwAESExO58MILadeuHeedd97Jdqb07NgBb78NnTtD1ao5E6PdQRljTC6SlJTEmDFjiI2N5bHHHgOgWrVqp7UzpeeFFyAlBQYNyqlILUEZY0yu8d///pcaNWrw6KOPUrNmTZ555hm/9tu8GcaOhQcegEqVghriaSxBGWNMLvDmm29y2223kZSUxPTp0/n666+pUaOGX/s+9xzkyQN+5rOAsTYoY4yJUgcOHGDfvn1UrVqVu+66i8TERB555BEKFCjg9zE2bIAJE+DRR6FcZlPOBoHdQRljTJTxbWfq0qULqkrZsmXp06dPtpITwNChULAg9O8fpGAzYQnKGGOiSOrzTI8++ii1atXi3XffRc5yNNc1a2DqVHf3FBMT4ED9YAnKGGOixKeffkqLFi1ISkpixowZzJ07N93nmfzx559wzz1QrBg89VRg4/SXJShjjIlgBw4cYNmyZQC0adOGt99+mzVr1tCmTZuzvnM6ehRatYK1a2HaNChdOpAR+88SlDHGRKCkpCRGjx5NlSpVaN++PcnJyRQsWJAePXpku53JV0ICtG0LixfDRx9B8+YBDDqbLEEZY0yEWbJkCdWrV+exxx6jdu3afPHFF+TLd+6dspOToWNHmDvXPff0978HINhzYN3MjTEmgnz77bf079+fKlWqMGPGDG6//fazrspL68kn4fPP4bXXoFu3gBzynNgdlDHGhLn9+/fz1VdfAXDDDTcwcODAc25nSuvrr+GNN+Cxx+DxxwNyyHNmCcoYY8JUUlISb7zxBrGxsbRv357Dhw8jItx6663n1M6U1l9/wX33weWXuxHLw4UlKGOMCUOzZ8+mevXqPP7449SpU4dFixZRrFixoJzriSfcaOUTJkDhwkE5xVmxNihjjAkz69evp2XLlkFpZ0rryy9h3DgYOBAaNAjKKc6a3UEZY0wY2L9/P1OnTgXg8ssvZ/bs2QFvZzrznNC9u5sdNyen0fCXJShjjAkh33amrl27smPHDgBatGgR0HamtHbtct3I9+2DiRPdeHvhxhKUMcaEyKxZs7jmmmtOtjOtXLmScjkwZPjs2VC9Oixd6qr3atYM+inPiiUoY4wJgb1799KuXTtSUlKYOXMmX331FVdffXVQz5mY6J51atkSLrwQli+He+8N6inPiSUoY4zJIfv37+eNN944Of3FN998w88//xzUThAAe/bAyy/DlVfCqFHQq5e7e6pWLWinDAhLUMYYE2RJSUm8/vrrxMbG0qdPH1avXg1Aw4YNg9bOFB8Ps2bBXXdB+fJuPqcKFVzZ6NFQqFBQThtQ1s3cGGOCRFWZPXs2TzzxBOvXr+fWW29l1KhRQavK27XLdRv/8ks3nt6xY24k8kcfhQcfdA/iRpKg3UGJSAURmS8ia0VkjYg87pWXEpG5IrLBey3plYuIvCEiG0XkJxGp7XOsLt72G0Ski095HRFZ7e3zhgTzHtkYY7IpPj6eBx54AFXliy++YM6cOQFPTseOuVHHb7vNTcnevTusWuXG0ps92z2AO3Jk5CUnCG4VXzLwpKpWAxoCj4hINaA/ME9VY4F53s8AtwGx3vIg8Da4hAYMBhoA9YHBqUnN26a7z34tgng9xhiTpX379jF48GASExMpXLgwX3/9NatXr6Z169YBbWfatAkeesh1dvjHP9zcTQMHwk8/webNMGYMtGgRnt3H/RW0Kj5V3Qns9N4fFpFfgHLAHUATb7MJQBzQzyufqKoKLBaREiJykbftXFU9ACAic4EWIhIHnK+qi73yicCdwOxgXZMxxmQkKSmJt956iyFDhnD48GFuvPFGmjZtSrUA90TYsAFeeukKvv4a8ueHDh2gSxe44QbIE2W9CnKkDUpEKgG1gCVAjJe8AHYBqTPdlwO2+ey23SvLrHx7OuXpnf9B3F0ZMTExxMXF+R37kSNHsrV9JLNrjU52rcGlqixevJi3336bbdu2UbduXXr27EmePHn8jiUlBZKThaSkPPz5ZwH27SvA/v0FOHCgIEeO5OPYsbwcO5aXAwcKsGRJafLlK0vbttvo0GEbpUsnArBgQRAvMkSCnqBEpCjwb6C3qh7yvcVVVRURDXYMqvoe8B5A3bp1tUmTJn7vGxcXR3a2j2R2rdEpWq5V1T3Hc/y4W3bvdt2njx07taxd+yuxsVVPbp+QAIcOnVoSEly5qksKqm6SvqSkU6+pxz9+3P2cJw/kywd587rX1CV/frfky5fCd9/1JTm5MLVrf0nRoi2ZMEFISeHkknq+lBR3DYcPw5EjbklIcOWZKVIEihWD4sXdwK7XXruYv/2tMVAh+L/4EApqghKR/LjkNEVVP/OKd4vIRaq606vC2+OV7+D033Z5r2wHp6oEU8vjvPLy6WxvjAmi5GTXhTkh4fTX9MoyW3f0qFt8E0JmS3x81l/kUDXd0tQv+EKFQMQlHRG3uCRz6rVwYShb1r0WKAAnTrglOdm9JiXBsWP72LDhRS68cCBQmosu+pyCBS8iX778JCefSmap58mT59T7AgVcLEWLuqVgwVPJrkABuOACuPhiuOgitxQv7o7nKy4uKRAfZdgLWoLyetR9APyiqqN8Vs0EugAvea8zfMp7icjHuA4Rf3lJbA4w3KdjRDNggKoeEJFDItIQV3XYGRgdrOsxJtwdO+a6Ge/a5e4udu92X+zr11fgu+/cF2zqXUJCgvsrPnWJj3flvktiYvqJJuskkbV8+eC881ziKFz49KV4cdfwn7a8cGGXYFLflyrlnu/xPc6yZd/RqFEjUitqChSA888/8wv+bCUlJfHmm28ydOhQDh8+zP/9X0Pat28PXBKYE5jTBPMOqjFwL7BaRH7wygbiEtM0Ebkf2ALc5a2bBbQENgLHgG4AXiJ6DljmbTcstcME0BMYDxTGdY6wDhImKiUluZ5ZGze6RvJNm+CPP04lpF27XHVR+i477afUv9SLFTu1FC7sygsVOv2v+UKF3F/4vq+BKAtUwkhr06ZELrwwOMf+z3/+w5NPPsn69etp1qwZo0aN4qqrrgrOyQwQ3F58C4GM+lTenM72CjySwbHGAePSKV8OBHfwKmNySEKCSzwbNrhEtHGj+3nTJtiyxVUvpSpa1N09XHgh1KnjXlOXmJhTr+edB99/v4CmTW842Y5izs7YsWNRVb788ktatmwZ1KGJjGMjSRiTA1Th4EHYvh22bXOvW7e6Zds2d3e0davbLlWpUnDppVCvnutKXKUKxMa65YILwN/vx8KFUyL6WZhQ2bdvH0OHDqVXr15cfvnljB07lvPPP5/8+fOHOrRcwxKUMVlITnZz5hw54hr1U199G+8TEk618Zw44danJqAtW1wSOnbs9OPmzevugipUgMaNoWvXUwkoNhZKlkw3HBNkiYmJvPXWWyfbmapXr87ll19O6dKlQx1armMJyhgf+/fD4sXwww+wZg38/DOsX+86DGRXTAxUrOhmK23ZEi65xCWk1KR04YVW5RZuZs2aRZ8+ffj1119p3rw5o0aNCviDtsZ/lqBMrrZ3L8ycCd9+6xLThg2n1lWsCFdf7ZJLxYqu3adIEfd63nluSe1RVrDg6c/KFCwY2UPM5Fbz5s1DRPjPf/7DbbfdZu1MIWYJyuQ6O3bAZ5+VY+hQ9/R9Sopr07n2Wrj/fmjYEGrXdr3bTHRLHTfv73//O02bNmXYsGG89NJL1s4UJixBmVzhyBH4/HOYOBHmzQPVWKpVc4Nr/v3vUKOG/50OTORLTEw8+TzTkSNHuPTSS2natClFihQJdWjGhyUoE3Xi411V3bp18MsvsHq1m3bg6FGoXBkGDYJLL11K5871Qx2qCYE5c+bw2GOP8euvv9rzTGHOEpSJeMeOwfffQ1ycW5YscQ+2grsrqlgROnZ0Iz43buzK4uKOZXZIE8U2bNhg7UwRwhKUiTgnTsDKlW7G0Llz4bvvXC+7vHmhbl3o0wdq1YIrr3Tdtc87L9QRm1Dau3cvgwcPpl69enTr1o0ePXrw0EMPWTtTBLAEZSJCSgr8738wdSr8+9+uOzhAzZrw+OPQtKm7O7KODSZVYmIiY8aMYdiwYRw5coQLLrgAgHz57GsvUtgnZcKSKvz+u5u6etEimDbN9b4rUgTatIHbb4ebb3a974xJ65tvvqFHjx5s2LDBnmeKYJagTNhISYHPPoO334YVK+Cvv1x5/vxu6uoRI1xiso5WJivHjx8nb968/Oc//6Fly5ahDsecJUtQJuRUYfp0GDIEfvrJjTnXsaNrR6pd2z0sW6hQqKM04Sy1neno0aM0adKEli1b0rx5c6vOi3D26Zkcd/iw6/69dq1bvvoKfvzRdWiYNMklJxsCyPgjbTtTu3btABARS05RwD5BE1TJyS4Zff/9qWX9+lPrCxRwd0gTJsA//uGGCTLGH4sWLaJr165s3LiRFi1aMGrUKHbv3h3qsEwA2deBCZg9e1yHhpUrXVL65Rf3wGzqM0llyrjhhO691yWlatXcg7OWlEx2pKSkkCdPHooVK0bBggWZNWsWt912G4AlqCiT4VeDiLymqr1F5AtA065X1TZBjcyEPVWYP99Vyy1adGqg1Tx54LLL3HNIt9/uklHDhq7Mnok0Z2vv3r0MGjSIo0ePMnHiRKpXr87q1avtQdsoltnfrpO81xE5EYiJHKpu6KDnn3dVdqVKwXXXwQMPuNfata1TgwmcxMRERo8ezbBhwzh69Ci9evU6eRdlySm6ZZigVHWF9/ptzoVjwt3cudC/v6vGu+QSeOst6NbNEpIJjpUrV3L33XezceNGWrZsyYgRI7jyyitDHZbJIVnW/otIY2AIUNHbXgBV1UuDG5oJJ8eOwdNPw5tvumnIP/gAOnVynRyMCbSkpCTy589PuXLlKF26NKNHj6ZFixahDsvkMH+apz8A+gArgBPBDceEo5Ur4Z573OjgffrA8OF2x2SCY8+ePQwaNIg1a9awYMECYmJiWLx4cajDMiGSx49t/lLV2aq6R1X3py5Bj8yE3M6dbr6kBg3g0CFXvTdqlCUnE3iJiYmMHDmS2NhYxo4dS+3atUlISAh1WCbE/LmDmi8irwKfASf/xajqyqBFZUJq2TJ4/XU3/l1yMnToAGPGuM4QxgTa+vXrad26tbUzmTP4k6AaeK91fcoUaBr4cEwo7dhRiKZNXdfxYsWgZ0/o1csNPWRMoB0/fpzChQtTsWJFYmNjrZ3JnCHLBKWqN+VEICZ0UlLgnXfgySfrUbAg/N//wX33wfnnhzoyE4327t3Ls88+y9y5c/n5558pXLgws2bNCnVYJgxl9qBuJ1WdLCJPpLdeVUcFLyyTU7Zscc8vff011K37F599VooKFUIdlYlGCQkJjB49mueee45jx47xyCOPkJycHOqwTBjL7A4qdVIDmwIuCv34o7tTmjrVdRV/5x2oWvUnKlRoEurQTBTasWMHN954I5s2baJly5aMHDmSK664ItRhmTCX2YO673qvQ3MuHBNsc+fCSy/BN9+4qdAfegj69oWKFSEuLtTRmWjz559/UqJECS6++GKuv/56xowZY+1Mxm/+PKj7RmbrVfWxwIVjgiU52Y0AMXIklC8PL78M3btDyZKhjsxEoz179vDss88ybdo01q1bR0xMDB9++GGowzIRxp9efIWAasAn3s/tgbXA98EKygTWvn1w993urqlnT1e1ZyNAmGBISEjgjTfe4Pnnn+fYsWP06tWLggULhjosE6H8SVDVgetUNRlARN4B/qeqPTLbSUTGAa2BPap6tVc2BOgO7PU2G6iqs7x1A4D7caNVPKaqc7zyFsDrQF5grKq+5JVXBj4GSuNGubhXVRP9vO5cY+VKaNsWdu+GDz+Erl1DHZGJVocPH6Z27dps3LiRVq1aMWLECGtnMufEn5EkSgK+HY6LemVZGQ+kV9n8f6pa01tSk1M1oANwlbfPWyKSV0TyAm8Ct+Hu4jp62wK87B2rCnAQl9yMjylToHFjN/r4woWWnExw7Ny5E4BixYrRoUMH/vvf//Lll19acjLnzJ8E9RKwSkTGi8gEYCUwPKudVHUBcMDPOO4APlbVBFX9HdgI1PeWjar6m3d39DFwh7gx9psC//L2nwDc6ee5ol5ysuv40KmTG6Zo+XKoWzfr/YzJjj179vDQQw9RsWJFVq9eDcBzzz1H8+bNQxyZiRb+PKj7oYjM5tSIEv1Uddc5nLOXiHQGlgNPqupBoBzgOyLkdq8MYFua8ga4ar0/U6sd02x/BhF5EHgQICYmhrhsdFc7cuRItrYPtUOH8vHcc9VYvrwUbdtup2fPTaxdq6xdm/W+kXat58Ku9ewlJiby2WefMXnyZOLj42nbti2bN29m//7QD9GZWz7X3HKdqGrQFqAS8LPPzzG4tqQ8wAvAOK98DNDJZ7sPgHbeMtan/F5v2zK4O6vU8gq+58lsqVOnjmbH/Pnzs7V9KP3wg+qll6oWKKA6dmz294+kaz1Xdq1nJykpSa+++moFtHXr1rpu3bqAHTsQcsvnGm3XCSzXdL6v/ekkETCqujv1vYi8D3zp/bjDSzKpyntlZFC+HyghIvnU3UX5bp8rjR8PDz/sBnSNi4Nrrw11RCaabNy4kcsuu4x8+fLRs2dPLrvsMpo1axbqsEyU86cNKmBE5CKfH9sCP3vvZwIdRKSg1zsvFlgKLANiRaSyiBTAdaSY6WXc+bg7LIAuwIycuIZwEx/vnmfq1g0aNYJVqyw5mcDZvXs3Dz74IFWrVj05Xt7DDz9sycnkCL/uoETkOiBWXXtUWaCous4Mme3zEdAEKCMi24HBQBMRqYkbDX0z8BCAqq4RkWm456uSgUdU9YR3nF7AHFzV4DhVXeOdoh/wsYg8D6zCVQvmKvv3Q7Nmriv5wIEwbBjkzRvqqEw0SEhI4PXXX+f555/n+PHj9O7dm0aNGoU6LJPL+DOSxGDcVBuXAx8C+YHJQOPM9lPVjukUZ5hEVPUFXLtU2vJZwBlDHavqb7hefrlScrJ7+Pbnn2HGDGjTJtQRmWihqtx8880sWrSI22+/nREjRlC1atVQh2VyIX+q+NoCbYCjAKr6BzaAbMgNGADz5sHbb1tyMoGxevVqkpKSEBH69u3LV199xcyZMy05mZDxJ0Elem0+CiAiRbLY3gTZRx/BiBFu2KL77gt1NCbSpbYz1ahRg7FjxwJw5513cuutt4Y4MpPb+ZOgponIu7hec92Br4H3gxuWycgPP8D998N117kx9Yw5W/Hx8bz88svExsby4Ycf0rt3bzp2TK9m3pjQ8OdB3REicitwCNcONUhV5wY9MnOGP/904+qVKgWffmoDvppzc/fddzNz5kxat27NiBEjuPzyy0MdkjGn8aeTxBPAJ5aUQu/ll90MuN99BxdeGOpoTCT64YcfqFSpEiVKlKBfv3488sgj1mXchC1/qviKAV+JyP9EpJeIxAQ7KHOmXbvgjTegQwdo2DDU0ZhIs3v3brp3707t2rV55ZVXAGjUqJElJxPWskxQqjpUVa8CHgEuAr4Vka+DHpk5zUsvQUICDLX5jU02JCQk8NFHHxEbG8v48ePp06cPTz/9dKjDMsYv2RnqaA+wCzfM0AXBCcekZ9s21528a1eIjQ11NCaSPProo7z//vu0adOGV1991bqMm4jiTxtUT+AuoCzwKdBdVf0YG9sEynPPuddBg0Ibh4kMq1atokSJElSuXJmnnnqKqlWr0rdv31CHZUy2+dMGVQHorapXqeoQS045a+NGGDcOHnoILrkk1NGYcLZr1y4eeOAB6tSpw5AhQwCIjY2lrk0GZiJUhndQInK+qh4CXvV+LuW7XlX9nYzQnIMhQ1x38oEDQx2JCVfx8fG8/vrrvPDCC8THx/PEE0/wz3/+M9RhGXPOMqvimwq0BlbgRpEQn3UKXBrEuAywbBlMnQpPPWXdyk3Ghg8fznPPPUebNm0YMWIEsdZQaaJEhglKVVt7r5VzLhyTatMmaN0aKlSAfv1CHY0JN6tWrSIlJYU6derQu3dvbrjhBm655ZZQh2VMQGXZBiUi8/wpM4Gza5ebRuPECZgzx40cYQy4dqb777+fOnXqMGDAAABKlSplyclEpczaoAoB5+HmcyrJqSq+84FyORBbrnToENx2m0tS33wDV1wR6ohMOIiPj+e1117jhRdeICEhwdqZTK6QWRvUQ0Bv4GJcO1RqgjoEjAluWLlTQgLceaeb4+mLL6BBg1BHZMLF+PHjGTBggLUzmVwlszao14HXReRRVR2dgzHlWi+8APPnw6RJ0KJFqKMxobZq1Sr27NlD8+bNue+++7jiiito0qRJqMMyJsf4M5r5aBG5GqgGFPIpnxjMwHKbHTvcHE8dOkCnTqGOxoTSrl27eOaZZ/jwww+pXr06zZo1o0CBApacTK7jTyeJwcBob7kJeAU3w64JoGefdZ0ihg8PdSQmVOLj43nppZeIjY1l0qRJPPHEE3z77beISNY7GxOF/BlJoh1wM7BLVbsBNYDiQY0ql/nxRxg/Hh57DCpbp/5c6+uvv2bAgAHcfPPNrF27lhEjRlC8uP1XM7mXP4PFHlfVFBFJFpHzcYPGVghyXLmGKvTtCyVL2mgRudHKlStZs2YN9957L61atWLp0qXUq1cv1GEZExb8uYNaLiIlcNO8rwBWAt8HM6jcZM4c+PprNxBsyZKhjsbklJ07d3LfffdRt25dnn32WRITExERS07G+PBnPqieqvqnqr4D3Ap08ar6zDlKTnZ3T1WqwMMPhzoakxPi4+N58cUXqVq1KpMnT+bJJ5/kxx9/pECBAqEOzZiwk9mDurUzW6eqK4MTUu4xaRKsWQP//rcbENZEv3Xr1vHMM8+cfJ6pSpUqoQ7JmLCVWRvUyEzWKdA0wLHkKikp8MorUKsWtG0b6mhMMK1cuZJ58+bx1FNPUbNmTdauXcsVNkSIMVnK7EHdm3IykNxm1ixYtw6mTAHrRRyddu7cyTPPPMP48eMpW7Ys3bt3p0SJEpacjPGTPzPqdk6v3B7UPTevvuomIGzfPtSRmECLj49n1KhRDB8+nMTERPr27cszzzxjXcaNySZ/upn7disqhHsmaiVgCeosLV0KCxbAqFGQP3+oozGBduDAAYYPH06zZs145ZVXrJ3JmLPkz1BHj/r+7HU5/zhYAeUGI0dC8eLwwAOhjsQEyooVK5gyZQojR47k4osvZt26dZQvXz7UYRkT0fx5Diqto0CW4x2IyDgR2SMiP/uUlRKRuSKywXst6ZWLiLwhIhtF5CffHoQi0sXbfoOIdPEpryMiq7193pAIGQ/mt9/gX/+CHj2gWLFQR2POVerzTPXq1WPy5Mls3boVwJKTMQHgz1h8X4jITG/5ElgPfO7HsccDacfk7g/MU9VYYJ73M8BtQKy3PAi87Z27FDAYaADUBwanJjVvm+4++0XE+N+vvQZ587phjUzkio+PZ/jw4cTGxjJ58mT69u3Lhg0bqFixYqhDMyZq+NMGNcLnfTKwRVW3Z7WTqi4QkUppiu8AmnjvJwBxQD+vfKKqKrBYREqIyEXetnNV9QCAiMwFWohIHHC+qi72yicCdwKz/biekDlwAD74AO65By6+ONTRmHORkpLC22+/TbNmzXj11Ve57LLLQh2SMVHHnzaobwG8cfjyee9LpSaNbIpR1Z3e+11AjPe+HLDNZ7vtXllm5dvTKU+XiDyIuzMjJiaGuLg4vwM+cuRItrbPzKRJFTl2rDI33LCMuLijATlmIAXyWsPd2Vzr+vXr+fTTT3n66acpUKAAY8aMoXjx4mzbto1t27ZlfYAQsc81+uSW6/Snm/mDwDAgHkjBzayrwKXncmJVVRHRczlGNs71HvAeQN26dTU78+rExcUFZB6eI0egXTto1Qq6dQvP8dYCda2RIDvX6vs8U5kyZYiJiaFGjRrBDTCA7HONPrnlOv3pJPEUcLWqVlLVS1W1sqqebXLa7VXd4b3u8cp3cPoI6eW9sszKy6dTHrbeeQf273fzPpnIkJSUdLKdacqUKTz11FNs2LAhopKTMZHMnwS1CTgWoPPNBFJ74nUBZviUd/Z68zUE/vKqAucAzUSkpNc5ohkwx1t3SEQaer33OvscK+wcP+5my731VmjQINTRGH/lzZuX6dOn06xZM9auXcvLL79sD9sak4P86SQxAPhORJYACamFqpppPzQR+QjXyaGMiGzH9cZ7CZgmIvcDW4C7vM1nAS2Bjbhk2M07xwEReQ5Y5m03zKftqyeup2BhXOeIsO0gMXYs7N4N//xnqCMxWVm+fDmDBw9mwoQJlClThvnz51OkSJFQh2VMruRPgnoX+AZYjWuD8ouqdsxg1c3pbKvAIxkcZxwwLp3y5cDV/sYTKgkJblDY66+HG24IdTQmI3/88QcDBw5kwoQJXHDBBaxbt47rrrvOkpMxIeRPgsqvqk8EPZIoNWECbN8O485IsSYcqCrDhw/nxRdfJCkpiaeffppnnnmG888/P9ShGZPr+ZOgZns9+b7g9Cq+s+lmnqskJcGLL0L9+nDLLaGOxqRHRPjxxx/teSZjwpA/CSq1qm6AT9k5dzPPDT76CDZvhjfesCk1wsny5cvp3bs3U6dOpVq1akyePNlmtDUmDPkz5XvldBZLTn4YNw6uuAJatw51JAZcO1PXrl2pV68eW7duPTluniUnY8KTzQcVJLt2uSk1Bg2yu6dw8OqrrzJ06FCSkpLo168f119/PS1aRMTwjcbkWjYfVJB8/jmoutEjTGioKqmD3O/du5fmzZvzyiuvcNlll+WKYWKMiXQ2H1SQfPqpq9676qpQR5I7LVu2jN69ezN48GCaNWvGSy+9RJ48ZzO7jDEmVII2H1RutmcPfPutu3uy6r2c9ccff9ClSxfq16/Ppk2bOHrUDcprycmYyONPG9QXuF574BJaNWBaMIOKdJ9/Dikp0L59qCPJXUaPHk3//v1JTk6mX79+DBw40J5nMiaCBW0+qNzsX/+C2Fi45ppQRxL9VBVVJU+ePOTPn5+WLVvyyiuvULmy3eQbE+kyrPcQkSoi0lhVv/VZFgEVRcSeZszAvn0wf75V7+WEpUuXct111/H+++8D8NBDD/Hpp59acjImSmRWMf8acCid8kPeOpOO6dPhxAmr3gumHTt20LlzZxo0aMCmTZtOVuOJ/UVgTFTJrIovRlVXpy1U1dXpTOVuPP/6F1x6KdSsGepIotPYsWN5/PHHSU5Opn///gwYMMDamYyJUpndQZXIZF3hAMcRFQ4cgHnz3N2T/TEfOKpKYmIiABUqVKBFixb88ssvvPjii5acjIlimSWo5SLSPW2hiDwArAheSJFrxgxITraHcwMptZ1p0KBBADRv3px///vfXHqpjbZlTLTLLEH1BrqJSJyIjPSWb4H7gcdzJLoIM2kSVK4MdeqEOpLIt3379tPamapVqxbqkIwxOSzDNihV3Q00EpGbODUx4H9U9ZsciSzCrF3reu+9+KJV752rjz/+mPvvv/9kO9PAgQMpVqxYqMMyxuQwf4Y6mg/Mz4FYItqbb0LBgnD//aGOJDKpKkePHqVo0aJUr16dVq1a8fLLL1uXcWNyMRv/JQAOHYKJE+Huu6Fs2VBHE3mWLl1K48aNud/L7tWqVWPatGmWnIzJ5SxBBcDEiXDkCPTqFepIIovv80y//fYbzZs3D3VIxpgw4s9QRyYTqq56r359qFcv6+2NM2vWLNq3b8+JEycYMGAAAwYMsHYmY8xpLEGdo2++gXXr3F2UyZyqsn//fsqUKUO9evVo164dQ4YMsao8Y0y6rIrvHI0ZA2XK2NBGWVmyZAmNGzemVatWpKSkULZsWSZMmGDJyRiTIUtQ52DLFpg5E7p3h0KFQh1NeNq+fTv33nsvDRs25Pfff6dHjx6hDskYEyGsiu8cvPuue7Xv3PR999133HrrrZw4cYKBAwfSv39/a2cyxvjN7qDOwcyZcMstcMkloY4kfKgqW7duBaBOnTp069aNdevW8cILL1hyMsZkiyWos3TgAKxZAzfcEOpIwsfixYtp1KgRjRs35tixYxQsWJAxY8ZQqVKlUIdmjIlAlqDO0vffu9frrgttHOFg+/btdOrUiWuvvZbNmzczbNgwClmjnDHmHFkb1FlauBDy5bNnn9atW0ft2rVJSUmxdiZjTECF5A5KRDaLyGoR+UFElntlpURkrohs8F5LeuUiIm+IyEYR+UlEavscp4u3/QYR6ZKT17BoEdSuDeedl5NnDQ8pKSmsXbsWgMsvv5ynn37a2pmMMQEXyiq+m1S1pqrW9X7uD8xT1VhgnvczwG1ArLc8CLwNLqEBg4EGQH1gcGpSC7aEBFi2LHdW7y1ZsoRGjRpRv359du3ahYgwZMgQa2cyxgRcOLVB3QFM8N5PAO70KZ+ozmKghIhcBDQH5qrqAVU9CMwFWuREoCtXQnw8NG6cE2cLD9u2baNTp040bNiQLVu2MGbMGC644IJQh2WMiWKhaoNS4CsRUeBdVX0PiFHVnd76XUCM974csM1n3+1eWUblZxCRB3F3X8TExBAXF+d3oEeOHDlj+08+qQBcBiwiLi7J72OFu/SuFeDAgQPcc889nDhxgk6dOvGPf/yDwoULs2DBgpwPMkAyutZoZNcafXLLdYYqQV2nqjtE5AJgrois812pquolr4DwEuB7AHXr1tUmTZr4vW9cXBxpt3/tNahSBf72t+i6hfK91pSUFJYuXUrDhg0BGDVqFC1btqRixYohjDBw0vtco5Vda/TJLdcZkio+Vd3hve4BPse1Ie32qu7wXvd4m+8AKvjsXt4ry6g8qFRdB4lobn9KfZ6pUaNGrFmzBoCHH344apKTMSYy5HiCEpEiIlIs9T3QDPgZmAmk9sTrAszw3s8EOnu9+RoCf3lVgXOAZiJS0usc0cwrC6pff4V9+6Kz/WnPnj3cc889XHvttWzdupUPP/yQK6+8MtRhGWNyqVBU8cUAn4tI6vmnqup/RWQZME1E7ge2AHd5288CWgIbgWNANwBVPSAizwHLvO2GqeqBYAe/aJF7jbY7qGPHjtG9e3cSEhJ45pln6N+/P0WLFg11WMaYXCzHE5Sq/gbUSKd8P3BzOuUKPJLBscYB4wIdY2YWLoTSpeHyy3PyrMGRkpLC3LlzadasGeeddx5PPPEEXbt2tao8Y0xYCKdu5hFh0SJo1AjcDWDkSm1natGiBfPmzQPgxhtvtORkjAkblqCyYc8e1wYVydV727ZtO62dafz48TRt2jTUYRljzBlsLL5s+O479xqpHSRSUlK46aab2LFjB//85z/p16+ftTMZEwBJSUls376d+Pj4HDlf8eLF+eWXX3LkXIFUqFAhypcvT/78+f3a3hJUNixaBAULQt26WW8bLlJSUvjss8+44447yJ8/P2PHjqVy5cpWlWdMAG3fvp1ixYpRqVIlJAfq/w8fPhxx416qKvv372f79u1UrlzZr32sii8bFi50yalgwVBH4p/vv/+ea6+9lvbt2/PRRx8B0KRJE0tOxgRYfHw8pUuXzpHkFKlEhNKlS2frLtPuoLJh6FD3oG6427ZtG/369eOjjz7i4osvZvz48XTq1CnUYRkT1Sw5ZS27vyNLUNnQrFmoI/BPhw4dWLlyJc8++yxPP/20tTMZYyKSVfFFgZSUFCZPnsyBA+455bfffpt169YxbNgwS07GmHSNHz+eXr16AfDOO+8wceJEwE1CWrNmTWrVqsWmTZuYOnVqyGK0BBXhvvvuOxo2bMi9997LBx98AED16tWtnckY47cePXrQuXNnAKZPn067du1YtWoV27ZtC2mCsiq+CLV161b69evHxx9/zMUXX8zEiRO55557Qh2WMble797www+BPWbNmm4Whcxs3ryZ1q1b8/PPPwMwYsSIk9Ny1KhRg2+//Zbk5GTGjRtH/fr1T9t3yJAhFC1alGrVqvHaa6+RN29e5s2bx/Hjx/nll1+oWbMmXbp0oU+fPoG9sCxYgopQTz31FDNnzuTZZ5+lX79+FClSJNQhGWPC1LFjx/jhhx9YsGAB991338kkllbLli3p0aMHRYsWpW/fvsTFxTFixAi+/PLLHI7YsQQVIVJSUpgyZQoNGjSgatWqvPrqq7z66qtccskloQ7NGOMjqzudUOjYsSMAN9xwA4cOHeLPP/8MbUB+sjaoCJDaztS5c2fef/99AC655BJLTsaYk/Lly0dKSsrJn32fN0rbvTtSusRbggpjW7ZsoWPHjjRu3JgdO3YwceJEXn755VCHZYwJQzExMezZs4f9+/eTkJBwWrXcJ598AsDChQspXrw4xYsX9+uYxYoV4/Dhw0GJ1x9WxRfGRo8ezfTp0xk0aBBPP/20tTMZYzKUP39+Bg0aRP369SlXrhxXXHHFyXWFChWiVq1aJCUlMW6c/zMUVa9enbx581KjRg26du2a450kUNVctdSpU0ezY/78+dna/lycOHFCJ0yYoAsWLFBV1YMHD+qWLVty7Pw5ea2hZtcanUJ1rWvXrs3R8x06dMjvbW+88UZdtmxZEKPJnvR+V8ByTef72qr4wkRqO1OXLl1O/oVTokQJa2cyxuRalqBCbOvWrSfbmf744w8mTpx48oFbY4w5V3FxcdSNpCkYfFgbVIjNmDGDGTNmMHjwYJ566ilrZzLGGI8lqByWkpLCpEmTKFy4MHfddRc9evTgzjvvpEKFCqEOzRhjwopV8eWgRYsWUb9+fbp27XpyfKv8+fNbcjLGmHRYgsoBW7Zs4e677+a6665j586dTJo0ic8++yzUYRljTFizBJUDVq9ezRdffMHgwYP59ddf6dSpE3ny2K/eGBO+KlWqxL59+0Iag7VBBUFqO9OBAwfo06cPrVq14vfffycmJibUoRljcoGTzxFF+B/ClqACbOHChfTu3ZsVK1bQpEkTHn/8cfLkyWPJyZhcpEmTJmeU3XXXXfTs2ZNjx47RsmXLM9Z37dqVrl27sm/fPtq1a3fauri4uCzPuXnzZpo3b06DBg1YsWIF9evXZ/Xq1Rw/fpx27doxdOhQwN0ZdenShS+++IKkpCQ+/fRTrrjiCvbv30/Hjh3ZsWMH1157Le75WWfUqFEnn8984IEH6N27N5s3b6ZFixY0bNiQ7777jnr16tGtWzcGDx7Mnj17mDJlyhnTemRXZKfXMLJt2zbuvvturr/+enbt2sXkyZOZN29exP8FY4yJHBs2bKBnz56sWbOGkSNHsnz5cn766Se+/fZbfvrpp5PblSlThpUrV/Lwww8zYsQIAIYOHcp1113HmjVraNu2LVu3bgVgxYoVfPjhhyxZsoTFixfz/vvvs2rVKgA2btzIk08+ybp161i3bh1Tp05l4cKFjBgxguHDh5/z9dgdVIAcPHiQWbNmMWTIEPr27WvPMxmTi2V2x3Peeedlur5MmTJ+3TGlp2LFijRs2BCAadOm8d5775GcnMzOnTtZu3Yt1atXB+Bvf/sbAHXq1DnZYWvBggUn37dq1YqSJUsCrlaobdu2J7/T/va3v/G///2PNm3aULlyZa655hoArrrqKm6++WZEhGuuuYbNmzef1TX4sgR1llJSUpg4cSKrV69m5MiRVK9ene3bt/s9SrAxxgRaahL5/fffGTFiBMuWLaNkyZJ07dr1tOk3ChYsCEDevHlJTk4+6/OlHgcgT548J3/OkyfPOR335DHP+Qi50MKFC6lfvz7dunXj+++/P/nBW3IyxoSDQ4cOUaRIEYoXL87u3buZPXt2lvvccMMNJ5/PnD17NgcPHgTg+uuvZ/r06Rw7doyjR4/y+eefc/311wc1/lR2B5UNf/zxB3369GHatGmUL1+eKVOm0LFjx4iZ/MsYkzvUqFGDWrVqccUVV1ChQgUaN26c5T6DBw+mY8eOXHXVVTRq1OjkQNW1a9ema9euJzs8PPDAA9SqVSsgVXhZSm+I80hagBbAemAj0D+r7c9luo2dO3dqTEyMDhkyRI8ePZqt40QCm5YhOtm1Bl84T7cRbrIz3UZE30GJSF7gTeBWYDuwTERmquraYJzvwgsvZPPmzRQqVCgYhzfGGOMj0tug6gMbVfU3VU0EPgbuCOYJLTkZY0zOEPV5GCvSiEg7oIWqPuD9fC/QQFV7pdnuQeBBgJiYmDoff/yx3+c4cuQIRYsWDVzQYcyuNTrZtQZf8eLFueyyy3KsPfrEiRPkzZs3R84VSKrKpk2b+Ouvv04rv+mmm1ao6hmTVkV0FZ+/VPU94D2AunXranpPeWckLi4u3afCo5Fda3Syaw2+33//ncTEREqXLp0jSerw4cMUK1Ys6OcJJFVl//79lChRglq1avm1T6QnqB2A71wV5b0yY4zJMeXLl2f79u3s3bs3R84XHx8fkc0NhQoVonz58n5vH+kJahkQKyKVcYmpA/CP0IZkjMlt8ufPT+XKlXPsfHFxcX7fhUSyiE5QqposIr2AOUBeYJyqrglxWMYYYwIgohMUgKrOAmaFOg5jjDGBFendzI0xxkSpiO5mfjZEZC+wJRu7lAFCO61kzrFrjU52rdEn2q6zoqqWTVuY6xJUdonI8vT650cju9boZNcafXLLdVoVnzHGmLBkCcoYY0xYsgSVtfdCHUAOsmuNTnat0SdXXKe1QRljjAlLdgdljDEmLFmCMsYYE5YsQXlEpIWIrBeRjSLSP531BUXkE2/9EhGpFIIwA8KPa+0qIntF5AdveSAUcZ4rERknIntE5OcM1ouIvOH9Hn4Skdo5HWOg+HGtTUTkL5/PdFBOxxgIIlJBROaLyFoRWSMij6ezTVR8rn5ea1R8rhlKb5rd3LbgxvHbBFwKFAB+BKql2aYn8I73vgPwSajjDuK1dgXGhDrWAFzrDUBt4OcM1rcEZgMCNASWhDrmIF5rE+DLUMcZgOu8CKjtvS8G/JrOv9+o+Fz9vNao+FwzWuwOyvFnZt47gAne+38BN0tOzU4WWDk+C3GoqOoC4EAmm9wBTFRnMVBCRC7KmegCy49rjQqqulNVV3rvDwO/AOXSbBYVn6uf1xrVLEE55YBtPj9v58x/CCe3UdVk4C+gdI5EF1j+XCvA373qkX+JSIV01kcDf38X0eJaEflRRGaLyFWhDuZcedXstYAlaVZF3eeaybVClH2uvixBmfR8AVRS1erAXE7dOZrItRI33lkNYDQwPbThnBsRKQr8G+itqodCHU8wZXGtUfW5pmUJyvFnZt6T24hIPqA4sD9HogusLK9VVferaoL341igTg7FltNyzYzMqnpIVY9472cB+UWkTIjDOisikh/3hT1FVT9LZ5Oo+VyzutZo+lzTYwnKOTkzr4gUwHWCmJlmm5lAF+99O+Ab9VopI0yW15qmvr4Nru47Gs0EOnu9vhoCf6nqzlAHFQwicmFqm6mI1Mf934+4P7C8a/gA+EVVR2WwWVR8rv5ca7R8rhmJ+AkLA0EzmJlXRIYBy1V1Ju4fyiQR2YhrjO4QuojPnp/X+piItAGScdfaNWQBnwMR+QjXy6mMiGwHBgP5AVT1HdxEly2BjcAxoFtoIj13flxrO+BhEUkGjgMdIvQPrMbAvcBqEfnBKxsIXAJR97n6c63R8rmmy4Y6MsYYE5asis8YY0xYsgRljDEmLFmCMsYYE5YsQRljjAlLlqCMMcaclawGKU5n+7t8Br+dmtX2lqCM8SEiKiIjfX7uKyJDAnTs8SLSLkDHWuKNXr1VTh95vlIgjp/O+QaexT5dRWRMMOIxYWM80MKfDUUkFhgANFbVq4DeWe1jCcqY0yUAfwu3p/G90UtOUtUGqloTGIQbWb+mt2zOznGyIdsJykS/9AYpFpHLROS/IrJCRP4nIld4q7oDb6rqQW/fPVkd3xKUMadLBt4D+qRdkfYOSESOeK9NRORbEZkhIr+JyEsico+ILBWR1SJymc9hbhGR5SLyq4i09vbPKyKvisgyb4Deh3yO+z8RmQmszSpwEbndu7NaJSJfi0iMVz5ERCaJyCLcw+ZlRWSuV80yVkS2pCZkEenkxf2DiLzrxfYSUNgrm5LRdl55N+/aluIeNDW5z3vAo6paB+gLvOWVVwWqisgiEVksIlneeVmCMuZMbwL3iEjxbOxTA+gBXIl7+r+qqtbHjWX4qM92lXBTnrQC3hGRQsD9uOF46gH1gO4iUtnbvjbwuKpW9SOGhUBDVa2Fm0blaZ911YBbVLUjbpSJb7xqln/hjUwgIlcCd+OqYGoCJ4B7VLU/cNy7Q7sno+3EDZE1FJeYrvPOaXIRcQPbNgI+9Ua/eBc3rxW4kYticSOedATeF5ESmR3PhjoyJg1VPSQiE4HHcMPH+GNZ6nhvIrIJ+MorXw3c5LPdNFVNATaIyG/AFUAzoLrP3Vlx3H/kRGCpqv7uZwzlgU+8RFEA8N1vpqqmXst1QFvvWv8rIge98ptxAwMv84Z3KwykVw2T0XYNgDhV3ev9Hj7B/dVsco88wJ/eHy5pbcdNHpkE/C4iv+L+nS/L7GDGmDO9hruzKeJTloz3f0ZE8uCSQKoEn/cpPj+ncPofgmnHFlPczK+P+rQjVVbV1AR3NBsxj8bNhHwN8BBQyGedP8cRYIJPHJer6pBz2M7kMt50IL+LSHtwA96KSA1v9XTc3RNelXJV4LfMjmcJyph0qOoBYBouSaXazKmpR9rgDcaaTe1FJI/XLnUpsB43cO/D4qZWQESqikiRzA6SgeKcmlaiSybbLQLu8s7VDCjplc8D2onIBd66UiJS0VuXlBpfJtstAW4UkdLetu3P4hpMBBE3SPH3wOUisl1E7gfuAe4XkR+BNZyasXsOsF9E1gLzgadUNdOR162Kz5iMjQR6+fz8PjDD+4/3X7J3d5NqK7AUOB/ooarxIjIW1za1Ulyd2V7gzrM49hBc3f9B4BugcgbbDQU+EpF7cV8uu4DDqrpPRP4JfOXdISYBjwBbcA3fP4nISq8d6oztVHWxuC753wN/Aj+cxTWYCOK1aabnjA4Q3ijrT3iLX2w0c2NyGREpCJzwpl65Fng7gzYDY0LK7qCMyX0uAaZ5dz+JuOdTjAk7dgdljDEmLFknCWOMMWHJEpQxxpiwZAnKGGNMWLIEZYwxJixZgjLGGBOW/h82xquqOXrTtQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "perf_lgwum.plot_uplift(metric=\"qini\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion\n",
    "From the Qini curves, you can see the Uplift DRF algorithm performs better than the LGWUM algorithm. The main reason is, that the split in Uplift DRF can be more precious thanks to information about both treatment and control groups."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Uplift trees modeling sources:\n",
    "\n",
    "- N. J. Radcliffe, and P. D. Surry, “Real-World Uplift Modelling withSignificance-Based Uplift Trees”, Stochastic Solutions White Paper, 2011.\n",
    "\n",
    "- P. D. Surry, and N. J. Radcliffe, “Quality measures for uplift models”, 2011.\n",
    "\n",
    "## References\n",
    "\n",
    "- P. Rzepakowski, and S. Jaroszewicz, “Decision trees for uplift modeling with single and multiple treatments”, 2012.\n",
    "\n",
    "- Hugh Huyton, “Criteo Uplift Modelling“, 2021, https://www.kaggle.com/code/hughhuyton/criteo-uplift-modelling/notebook.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
